Methods, apparatus and systems for surface type detection in connection with locate and marking operations

ABSTRACT

Systems, methods, and apparatus for performing surface type detection in connection with locate and marking operations. In some embodiments, one or more sensors (e.g., radiation sensors, acoustic sensors, color sensors, light sensors, etc.) may be employed to collect information regarding a surface, such as a ground surface on which marking material is to be dispensed to mark the presence or absence of an underground facility. The collected sensor data may be analyzed to provide an estimate of a type of the surface that is being sensed. For example, a still-image or video camera may be used as a sensor that detects visible light reflecting from a surface. One or more images of the surface captured by the camera may be analyzed using some suitable image analysis software to identify one or more characteristics (e.g., color, intensity, randomness, presence/absence of lines, etc.) that may be indicative of a surface type. As another example, one or more radiation sensors may be employed to measure an amount of electromagnetic radiation reflected by the sensed surface one or more selected wavelengths or ranges of wavelengths to identify a spectral signature that may also be indicative of a surface type.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims a priority benefit, under 35 U.S.C. §119(e), toU.S. provisional patent application Ser. No. 61/374,034, filed on Aug.16, 2010, entitled “Methods and Apparatus for Surface Type Detection inConnection with Locate and Marking Operations.”

This application also claims a priority benefit, under 35 U.S.C.§119(e), to U.S. provisional patent application Ser. No. 61/373,451,filed on Aug. 13, 2010, entitled “Methods and Apparatus for Surface TypeDetection in Connection with Locate and Marking Operations.”

This application also claims a priority benefit, under 35 U.S.C. §120,as a continuation-in-part (CIP) of U.S. non-provisional patentapplication Ser. No. 13/210,237, filed on Aug. 15, 2011, entitled“Methods, Apparatus and Systems for Marking Material Color Detection inConnection with Locate and Marking Operations.”

Ser. No. 13/210,237 in turn claims a priority benefit, under 35 U.S.C.§119(e), to U.S. provisional patent application Ser. No. 61/373,475,filed on Aug. 13, 2010, entitled “Methods and Apparatus for MarkingMaterial Color Detection in Connection with Locate and MarkingOperations.”

Each of the above-identified applications is hereby incorporated hereinby reference in its entirety.

BACKGROUND

Field service operations may be any operation in which companiesdispatch technicians and/or other staff to perform certain activities,for example, installations, services and/or repairs. Field serviceoperations may exist in various industries, examples of which include,but are not limited to, network installations, utility installations,security systems, construction, medical equipment, heating, ventilatingand air conditioning (HVAC) and the like.

An example of a field service operation in the construction industry isa so-called “locate and marking operation,” also commonly referred tomore simply as a “locate operation” (or sometimes merely as “a locate”).In a typical locate operation, a locate technician visits a work site inwhich there is a plan to disturb the ground (e.g., excavate, dig one ormore holes and/or trenches, bore, etc.) so as to determine a presence oran absence of one or more underground facilities (such as various typesof utility cables and pipes) in a dig area to be excavated or disturbedat the work site. In some instances, a locate operation may be requestedfor a “design” project, in which there may be no immediate plan toexcavate or otherwise disturb the ground, but nonetheless informationabout a presence or absence of one or more underground facilities at awork site may be valuable to inform a planning, permitting and/orengineering design phase of a future construction project.

In many states, an excavator who plans to disturb ground at a work siteis required by law to notify any potentially affected undergroundfacility owners prior to undertaking an excavation activity. Advancednotice of excavation activities may be provided by an excavator (oranother party) by contacting a “one-call center.” One-call centerstypically are operated by a consortium of underground facility ownersfor the purposes of receiving excavation notices and in turn notifyingfacility owners and/or their agents of a plan to excavate. As part of anadvanced notification, excavators typically provide to the one-callcenter various information relating to the planned activity, including alocation (e.g., address) of the work site and a description of the digarea to be excavated or otherwise disturbed at the work site.

FIG. 1 illustrates an example in which a locate operation is initiatedas a result of an excavator 3110 providing an excavation notice to aone-call center 3120. An excavation notice also is commonly referred toas a “locate request,” and may be provided by the excavator to theone-call center via an electronic mail message, information entry via awebsite maintained by the one-call center, or a telephone conversationbetween the excavator and a human operator at the one-call center. Thelocate request may include an address or some other location-relatedinformation describing the geographic location of a work site at whichthe excavation is to be performed, as well as a description of the digarea (e.g., a text description), such as its location relative tocertain landmarks and/or its approximate dimensions, within which thereis a plan to disturb the ground at the work site. One-call centerssimilarly may receive locate requests for design projects (for which, asdiscussed above, there may be no immediate plan to excavate or otherwisedisturb the ground).

Once facilities implicated by the locate request are identified by aone-call center (e.g., via a polygon map/buffer zone process), theone-call center generates a “locate request ticket” (also known as a“locate ticket,” or simply a “ticket”). The locate request ticketessentially constitutes an instruction to inspect a work site andtypically identifies the work site of the proposed excavation or designand a description of the dig area, typically lists on the ticket all ofthe underground facilities that may be present at the work site (e.g.,by providing a member code for the facility owner whose polygon fallswithin a given buffer zone), and may also include various otherinformation relevant to the proposed excavation or design (e.g., thename of the excavation company, a name of a property owner or partycontracting the excavation company to perform the excavation, etc.). Theone-call center sends the ticket to one or more underground facilityowners 3140 and/or one or more locate service providers 3130 (who may beacting as contracted agents of the facility owners) so that they canconduct a locate and marking operation to verify a presence or absenceof the underground facilities in the dig area. For example, in someinstances, a given underground facility owner 3140 may operate its ownfleet of locate technicians (e.g., locate technician 3145), in whichcase the one-call center 3120 may send the ticket to the undergroundfacility owner 3140. In other instances, a given facility owner maycontract with a locate service provider to receive locate requesttickets and perform a locate and marking operation in response toreceived tickets on their behalf.

Upon receiving the locate request, a locate service provider or afacility owner (hereafter referred to as a “ticket recipient”) maydispatch a locate technician (e.g., locate technician 3150) to the worksite of planned excavation to determine a presence or absence of one ormore underground facilities in the dig area to be excavated or otherwisedisturbed. A typical first step for the locate technician includesutilizing an underground facility “locate device,” which is aninstrument or set of instruments (also referred to commonly as a “locateset”) for detecting facilities that are concealed in some manner, suchas cables and pipes that are located underground. The locate device isemployed by the technician to verify the presence or absence ofunderground facilities indicated in the locate request ticket aspotentially present in the dig area (e.g., via the facility owner membercodes listed in the ticket). This process is often referred to as a“locate operation.”

In one example of a locate operation, an underground facility locatedevice is used to detect electromagnetic fields that are generated by anapplied signal provided along a length of a target facility to beidentified. In this example, a locate device may include both a signaltransmitter to provide the applied signal (e.g., which is coupled by thelocate technician to a tracer wire disposed along a length of afacility), and a signal receiver which is generally a hand-heldapparatus carried by the locate technician as the technician walksaround the dig area to search for underground facilities. FIG. 2illustrates a conventional locate device 3500 (indicated by the dashedbox) that includes a transmitter 3505 and a locate receiver 3510. Thetransmitter 3505 is connected, via a connection point 3525, to a targetobject (in this example, underground facility 3515) located in theground 3520. The transmitter generates the applied signal 3530, which iscoupled to the underground facility via the connection point (e.g., to atracer wire along the facility), resulting in the generation of amagnetic field 3535. The magnetic field in turn is detected by thelocate receiver 3510, which itself may include one or more detectionantenna (not shown). The locate receiver 3510 indicates a presence of afacility when it detects electromagnetic fields arising from the appliedsignal 3530. Conversely, the absence of a signal detected by the locatereceiver generally indicates the absence of the target facility.

In yet another example, a locate device employed for a locate operationmay include a single instrument, similar in some respects to aconventional metal detector. In particular, such an instrument mayinclude an oscillator to generate an alternating current that passesthrough a coil, which in turn produces a first magnetic field. If apiece of electrically conductive metal is in close proximity to the coil(e.g., if an underground facility having a metal component is below/nearthe coil of the instrument), eddy currents are induced in the metal andthe metal produces its own magnetic field, which in turn affects thefirst magnetic field. The instrument may include a second coil tomeasure changes to the first magnetic field, thereby facilitatingdetection of metallic objects.

In addition to the locate operation, the locate technician alsogenerally performs a “marking operation,” in which the technician marksthe presence (and in some cases the absence) of a given undergroundfacility in the dig area based on the various signals detected (or notdetected) during the locate operation. For this purpose, the locatetechnician conventionally utilizes a “marking device” to dispense amarking material on, for example, the ground, pavement, or other surfacealong a detected underground facility. Marking material may be anymaterial, substance, compound, and/or element, used or which may be usedseparately or in combination to mark, signify, and/or indicate. Examplesof marking materials may include, but are not limited to, paint, chalk,dye, and/or iron. Marking devices, such as paint marking wands and/orpaint marking wheels, provide a convenient method of dispensing markingmaterials onto surfaces, such as onto the surface of the ground orpavement.

FIGS. 3A and 3B illustrate a conventional marking device 50 with amechanical actuation system to dispense paint as a marker. Generallyspeaking, the marking device 50 includes a handle 38 at a proximal endof an elongated shaft 36 and resembles a sort of “walking stick,” suchthat a technician may operate the marking device while standing/walkingin an upright or substantially upright position. A marking dispenserholder 40 is coupled to a distal end of the shaft 36 so as to containand support a marking dispenser 56, e.g., an aerosol paint can having aspray nozzle 54. Typically, a marking dispenser in the form of anaerosol paint can is placed into the holder 40 upside down, such thatthe spray nozzle 54 is proximate to the distal end of the shaft (closeto the ground, pavement or other surface on which markers are to bedispensed).

In FIGS. 3A and 3B, the mechanical actuation system of the markingdevice 50 includes an actuator or mechanical trigger 42 proximate to thehandle 38 that is actuated/triggered by the technician (e.g, viapulling, depressing or squeezing with fingers/hand). The actuator 42 isconnected to a mechanical coupler 52 (e.g., a rod) disposed inside andalong a length of the elongated shaft 36. The coupler 52 is in turnconnected to an actuation mechanism 58, at the distal end of the shaft36, which mechanism extends outward from the shaft in the direction ofthe spray nozzle 54. Thus, the actuator 42, the mechanical coupler 52,and the actuation mechanism 58 constitute the mechanical actuationsystem of the marking device 50.

FIG. 3A shows the mechanical actuation system of the conventionalmarking device 50 in the non-actuated state, wherein the actuator 42 is“at rest” (not being pulled) and, as a result, the actuation mechanism58 is not in contact with the spray nozzle 54. FIG. 3B shows the markingdevice 50 in the actuated state, wherein the actuator 42 is beingactuated (pulled, depressed, squeezed) by the technician. When actuated,the actuator 42 displaces the mechanical coupler 52 and the actuationmechanism 58 such that the actuation mechanism contacts and appliespressure to the spray nozzle 54, thus causing the spray nozzle todeflect slightly and dispense paint. The mechanical actuation system isspring-loaded so that it automatically returns to the non-actuated state(FIG. 3A) when the actuator 42 is released.

In some environments, arrows, flags, darts, or other types of physicalmarks may be used to mark the presence or absence of an undergroundfacility in a dig area, in addition to or as an alternative to amaterial applied to the ground (such as paint, chalk, dye, tape) alongthe path of a detected utility. The marks resulting from any of a widevariety of materials and/or objects used to indicate a presence orabsence of underground facilities generally are referred to as “locatemarks.” Often, different color materials and/or physical objects may beused for locate marks, wherein different colors correspond to differentutility types. For example, the American Public Works Association (APWA)has established a standardized color-coding system for utilityidentification for use by public agencies, utilities, contractors andvarious groups involved in ground excavation (e.g., red=electric powerlines and cables; blue=potable water; orange=telecommunication lines;yellow=gas, oil, steam). In some cases, the technician also may provideone or more marks to indicate that no facility was found in the dig area(sometimes referred to as a “clear”).

As mentioned above, the foregoing activity of identifying and marking apresence or absence of one or more underground facilities generally isreferred to for completeness as a “locate and marking operation.”However, in light of common parlance adopted in the constructionindustry, and/or for the sake of brevity, one or both of the respectivelocate and marking functions may be referred to in some instances simplyas a “locate operation” or a “locate” (i.e., without making any specificreference to the marking function). Accordingly, it should beappreciated that any reference in the relevant arts to the task of alocate technician simply as a “locate operation” or a “locate” does notnecessarily exclude the marking portion of the overall process. At thesame time, in some contexts a locate operation is identified separatelyfrom a marking operation, wherein the former relates more specificallyto detection-related activities and the latter relates more specificallyto marking-related activities.

Inaccurate locating and/or marking of underground facilities can resultin physical damage to the facilities, property damage, and/or personalinjury during the excavation process that, in turn, can expose afacility owner or contractor to significant legal liability. Whenunderground facilities are damaged and/or when property damage orpersonal injury results from damaging an underground facility during anexcavation, the excavator may assert that the facility was notaccurately located and/or marked by a locate technician, while thelocate contractor who dispatched the technician may in turn assert thatthe facility was indeed properly located and marked. Proving whether theunderground facility was properly located and marked can be difficultafter the excavation (or after some damage, e.g., a gas explosion),because in many cases the physical locate marks (e.g., the markingmaterial or other physical marks used to mark the facility on thesurface of the dig area) will have been disturbed or destroyed duringthe excavation process (and/or damage resulting from excavation).

Previous efforts at documenting locate operations have focused primarilyon locate devices that employ electromagnetic fields to determine thepresence of an underground facility. For example, U.S. Pat. No.5,576,973, naming inventor Alan Haddy and entitled “Apparatus and Methodfor Obtaining Geographical Positional Data for an Object LocatedUnderground” (hereafter “Haddy”), is directed to a locate device (i.e.,a “locator”) that receives and stores data from a global positioningsystem (“GPS”) to identify the position of the locate device as anunderground object (e.g., a cable) is detected by the locate device.Haddy notes that by recording geographical position data relating to thedetected underground object, there is no need to physically mark thelocation of the underground object on the ground surface, and therecorded position data may be used in the future to re-locate theunderground object.

Similarly, U.S. Pat. No. 7,319,387, naming inventors Willson et al. andentitled “GPS Interface for Locating Device” (hereafter “Willson”), isdirected to a locate device for locating “position markers,” i.e.,passive antennas that reflect back RF signals and which are installedalong buried utilities. In Willson, a GPS device may be communicativelycoupled to the locate device, or alternatively provided as an integralpart of the locate device, to store GPS coordinate data associated withposition markers detected by the locate device. Electronic memory isprovided in the locate device for storing a data record of the GPScoordinate data, and the data record may be uploaded to a remotecomputer and used to update a mapping database for utilities.

U.S. Publication No. 2006/0282280, naming inventors Stotz et al. andentitled “Ticket and Data Management” (hereafter “Stotz”), also isdirected to a locate device (i.e., a “locator”) including a GPSreceiver. Upon detection of the presence of a utility line, Stotz'locate device can update ticket data with GPS coordinates for thedetected utility line. Once the locate device has updated the ticketdata, the reconfigured ticket data may be transmitted to a network.

U.S. Publication No. 2007/0219722, naming inventors Sawyer, Jr. et al.and entitled “System and Method for Collecting and Updating GeographicalData” (hereafter “Sawyer”), is directed to collecting and recording datarepresentative of the location and characteristics of utilities andinfrastructure in the field for creating a grid or map. Sawyer employs afield data collection unit including a “locating pole” that is placed ontop of or next to a utility to be identified and added to the grid ormap. The locating pole includes an antenna coupled to a locationdetermination system, such as a GPS unit, to provide longitudinal andlatitudinal coordinates of the utility under or next to the end of thelocating pole. The data gathered by the field data collection unit issent to a server to provide a permanent record that may be used fordamage prevention and asset management operations.

SUMMARY

Applicants have recognized and appreciated that uncertainties which maybe attendant to locate and marking operations may be significantlyreduced by collecting various information particularly relating to themarking operation, rather than merely focusing on information relatingto detection of underground facilities via a locate device. In manyinstances, excavators arriving to a work site have only physical locatemarks on which to rely to indicate a presence or absence of undergroundfacilities, and they are not generally privy to information that mayhave been collected previously during the locate operation. Accordingly,the integrity and accuracy of the physical locate marks applied during amarking operation arguably is significantly more important in connectionwith reducing risk of damage and/or injury during excavation than thelocation of where an underground facility was detected via a locatedevice during a locate operation.

Furthermore, Applicants have recognized and appreciated that thelocation at which an underground facility ultimately is detected duringa locate operation is not always where the technician physically marksthe ground, pavement or other surface during a marking operation; infact, technician imprecision or negligence, as well as various groundconditions and/or different operating conditions amongst differentlocate device, may in some instances result in significant discrepanciesbetween detected location and physical locate marks. Accordingly, havingdocumentation (e.g., an electronic record) of where physical locatemarks were actually dispensed (i.e., what an excavator encounters whenarriving to a work site) is notably more relevant to the assessment ofliability in the event of damage and/or injury than where an undergroundfacility was detected prior to marking.

Examples of marking devices configured to collect some types ofinformation relating specifically to marking operations are provided inU.S. publication no. 2008-0228294-A1, published Sep. 18, 2008, filedMar. 13, 2007, and entitled “Marking System and Method With Locationand/or Time Tracking,” and U.S. publication no. 2008-0245299-A1,published Oct. 9, 2008, filed Apr. 4, 2007, and entitled “Marking Systemand Method,” both of which publications are incorporated herein byreference. These publications describe, amongst other things, collectinginformation relating to the geographic location, time, and/orcharacteristics (e.g., color/type) of dispensed marking material from amarking device and generating an electronic record based on thiscollected information. Applicants have recognized and appreciated thatcollecting information relating to both geographic location and color ofdispensed marking material provides for automated correlation ofgeographic information for a locate mark to facility type (e.g.,red=electric power lines and cables; blue=potable water;orange=telecommunication lines; yellow=gas, oil, steam); in contrast, inconventional locate devices equipped with GPS capabilities as discussedabove, there is no apparent automated provision for readily linking GPSinformation for a detected facility to the type of facility detected.

Applicants have further appreciated that building a more comprehensiveelectronic record of information relating to marking operations furtherfacilitates ensuring the accuracy of such operations. For example,Applicants have recognized and appreciated that collecting and analyzinginformation relating to a type of surface being marked (e.g., dirt,grass, sand, gravel, asphalt, concrete, etc.) may facilitate ensuringaccuracy of locate and marking operations, for example, by ensuring thatan appropriate type of marking material is applied and/or by detectingundesirable operating conditions.

In view of the foregoing, various inventive embodiments disclosed hereinrelate generally to systems and methods for surface type detection inconnection with locate and marking operations.

In some embodiments, one or more sensors may be employed to collectinformation regarding a surface, such as a ground surface on whichmarking material is to be dispensed to mark the presence or absence ofan underground facility. The collected sensor data may be analyzed toprovide one or more estimates of a type of the surface that is beingsensed. For instance, based the sensor data, it may be determined thatthe surface being sensed is likely to be asphalt, concrete, wood, grass,dirt (or soil), brick, gravel, stone, snow, or any other surface type orcombination of surface types.

In some further embodiments, a combination of different sensing and/oranalysis techniques may be employed, which may lead to multiple surfacetype hypotheses for the sensed surface. These hypotheses may beaggregated and/or reconciled to further improve accuracy of surface typedetection.

In yet some further embodiments, some of the sensors used to collectdata from a surface may be attached to a marking device, so that sensordata may be collected from the surface as it is being marked (or shortlybefore or after it is marked). Each such sensor may be communicativelycoupled to one or more other components of the marking device that areconfigured to receive and process sensor data.

In summary, one embodiment of the present disclosure is directed to anapparatus for determining a surface type of a surface on which markingmaterial is to be dispensed by a marking device to mark a presence or anabsence of at least one underground facility within a dig area, whereinat least a portion of the dig area is planned to be excavated ordisturbed during excavation activities. The apparatus comprises: atleast one communication interface; at least one memory to storeprocessor-executable instructions; and at least one processorcommunicatively coupled to the at least one memory and the at least onecommunication interface. Upon execution of the processor-executableinstructions, the at least one processor: A) obtains sensor datarelating to the surface to be marked, the sensor data being collected byone or more sensors attached to the marking device; B) retrievesreference data associated with a plurality of surface types; and C)generates surface type information based at least in part on the sensordata and the reference data.

A further embodiment of the present disclosure is directed to a methodfor use in a system comprising at least one communication interface, atleast one memory to store processor-executable instructions, and atleast one processor communicatively coupled to the at least one memoryand the at least one communication interface. The method may beperformed for determining a surface type of a surface on which markingmaterial is to be dispensed by a marking device to mark a presence or anabsence of at least one underground facility within a dig area, whereinat least a portion of the dig area is planned to be excavated ordisturbed during excavation activities. The method comprises acts of: A)obtaining sensor data relating to the surface to be marked, the sensordata being collected by one or more sensors attached to the markingdevice; B) retrieving, from the at least one memory, reference dataassociated with a plurality of surface types; and C) using the at leastone processor to generate surface type information based at least inpart on the sensor data and the reference data.

Yet a further embodiment of the present disclosure is directed to atleast one non-transitory computer-readable storage medium encoded withat least one program including processor-executable instructions that,when executed by at least one processor, perform the above describedmethod for determining a surface type.

Yet a further embodiment of the present disclosure is directed to amarking apparatus for performing a marking operation to mark on asurface a presence or an absence of at least one underground facility.The marking apparatus comprises: at least one actuator to dispense amarking material so as to form at least one locate mark on the surfaceto mark the presence or the absence of the at least one undergroundfacility; at least one sensor for sensing the surface to be marked; atleast one user interface including at least one display device; at leastone communication interface; at least one memory to storeprocessor-executable instructions; and at least one processorcommunicatively coupled to the at least one memory, the at least onecommunication interface, the at least one user interface, and the atleast one actuator. Upon execution of the processor-executableinstructions, the at least one processor: A) obtains sensor datarelating to the surface to be marked, the sensor data being collected bythe at least one sensor; B) retrieves, from the at least one memory,reference data associated with a plurality of surface types; and C)generates surface type information based at least in part on the sensordata and the reference data.

For purposes of the present disclosure, the term “dig area” refers to aspecified area of a work site within which there is a plan to disturbthe ground (e.g., excavate, dig holes and/or trenches, bore, etc.), andbeyond which there is no plan to excavate in the immediate surroundings.Thus, the metes and bounds of a dig area are intended to providespecificity as to where some disturbance to the ground is planned at agiven work site. It should be appreciated that a given work site mayinclude multiple dig areas.

The term “facility” refers to one or more lines, cables, fibers,conduits, transmitters, receivers, or other physical objects orstructures capable of or used for carrying, transmitting, receiving,storing, and providing utilities, energy, data, substances, and/orservices, and/or any combination thereof. The term “undergroundfacility” means any facility beneath the surface of the ground. Examplesof facilities include, but are not limited to, oil, gas, water, sewer,power, telephone, data transmission, cable television (TV), and/orinternet services.

The term “locate device” refers to any apparatus and/or device fordetecting and/or inferring the presence or absence of any facility,including without limitation, any underground facility. In variousexamples, a locate device may include both a locate transmitter and alocate receiver (which in some instances may also be referred tocollectively as a “locate instrument set,” or simply “locate set”).

The term “marking device” refers to any apparatus, mechanism, or otherdevice that employs a marking dispenser for causing a marking materialand/or marking object to be dispensed, or any apparatus, mechanism, orother device for electronically indicating (e.g., logging in memory) alocation, such as a location of an underground facility. Additionally,the term “marking dispenser” refers to any apparatus, mechanism, orother device for dispensing and/or otherwise using, separately or incombination, a marking material and/or a marking object. An example of amarking dispenser may include, but is not limited to, a pressurized canof marking paint. The term “marking material” means any material,substance, compound, and/or element, used or which may be usedseparately or in combination to mark, signify, and/or indicate. Examplesof marking materials may include, but are not limited to, paint, chalk,dye, and/or iron. The term “marking object” means any object and/orobjects used or which may be used separately or in combination to mark,signify, and/or indicate. Examples of marking objects may include, butare not limited to, a flag, a dart, and arrow, and/or an RFID markingball. It is contemplated that marking material may include markingobjects. It is further contemplated that the terms “marking materials”or “marking objects” may be used interchangeably in accordance with thepresent disclosure.

The term “locate mark” means any mark, sign, and/or object employed toindicate the presence or absence of any underground facility. Examplesof locate marks may include, but are not limited to, marks made withmarking materials, marking objects, global positioning or otherinformation, and/or any other means. Locate marks may be represented inany form including, without limitation, physical, visible, electronic,and/or any combination thereof.

The terms “actuate” or “trigger” (verb form) are used interchangeably torefer to starting or causing any device, program, system, and/or anycombination thereof to work, operate, and/or function in response tosome type of signal or stimulus. Examples of actuation signals orstimuli may include, but are not limited to, any local or remote,physical, audible, inaudible, visual, non-visual, electronic,mechanical, electromechanical, biomechanical, biosensing or othersignal, instruction, or event. The terms “actuator” or “trigger” (nounform) are used interchangeably to refer to any method or device used togenerate one or more signals or stimuli to cause or causing actuation.Examples of an actuator/trigger may include, but are not limited to, anyform or combination of a lever, switch, program, processor, screen,microphone for capturing audible commands, and/or other device ormethod. An actuator/trigger may also include, but is not limited to, adevice, software, or program that responds to any movement and/orcondition of a user, such as, but not limited to, eye movement, brainactivity, heart rate, other data, and/or the like, and generates one ormore signals or stimuli in response thereto. In the case of a markingdevice or other marking mechanism (e.g., to physically or electronicallymark a facility or other feature), actuation may cause marking materialto be dispensed, as well as various data relating to the markingoperation (e.g., geographic location, time stamps, characteristics ofmaterial dispensed, etc.) to be logged in an electronic file stored inmemory. In the case of a locate device or other locate mechanism (e.g.,to physically locate a facility or other feature), actuation may cause adetected signal strength, signal frequency, depth, or other informationrelating to the locate operation to be logged in an electronic filestored in memory.

The terms “locate and marking operation,” “locate operation,” and“locate” generally are used interchangeably and refer to any activity todetect, infer, and/or mark the presence or absence of an undergroundfacility. In some contexts, the term “locate operation” is used to morespecifically refer to detection of one or more underground facilities,and the term “marking operation” is used to more specifically refer tousing a marking material and/or one or more marking objects to mark apresence or an absence of one or more underground facilities. The term“locate technician” refers to an individual performing a locateoperation. A locate and marking operation often is specified inconnection with a dig area, at least a portion of which may be excavatedor otherwise disturbed during excavation activities.

The terms “locate request” and “excavation notice” are usedinterchangeably to refer to any communication to request a locate andmarking operation. The term “locate request ticket” (or simply “ticket”)refers to any communication or instruction to perform a locateoperation. A ticket might specify, for example, the address ordescription of a dig area to be marked, the day and/or time that the digarea is to be marked, and/or whether the user is to mark the excavationarea for certain gas, water, sewer, power, telephone, cable television,and/or some other underground facility. The term “historical ticket”refers to past tickets that have been completed.

The term “user” refers to an individual utilizing a locate device and/ora marking device and may include, but is not limited to, land surveyors,locate technicians, and support personnel.

The following U.S. published applications are hereby incorporated hereinby reference:

U.S. Pat. No. 7,640,105, issued Dec. 29, 2009, filed Mar. 13, 2007, andentitled “Marking System and Method With Location and/or Time Tracking;”

U.S. publication no. 2010-0094553-A1, published Apr. 15, 2010, filedDec. 16, 2009, and entitled “Systems and Methods for Using Location Dataand/or Time Data to Electronically Display Dispensing of Markers by AMarking System or Marking Tool;”

U.S. publication no. 2008-0245299-A1, published Oct. 9, 2008, filed Apr.4, 2007, and entitled “Marking System and Method;”

U.S. publication no. 2009-0013928-A1, published Jan. 15, 2009, filedSep. 24, 2008, and entitled “Marking System and Method;”

U.S. publication no. 2010-0090858-A1, published Apr. 15, 2010, filedDec. 16, 2009, and entitled “Systems and Methods for Using MarkingInformation to Electronically Display Dispensing of Markers by a MarkingSystem or Marking Tool;”

U.S. publication no. 2009-0238414-A1, published Sep. 24, 2009, filedMar. 18, 2008, and entitled “Virtual White Lines for Delimiting PlannedExcavation Sites;”

U.S. publication no. 2009-0241045-A1, published Sep. 24, 2009, filedSep. 26, 2008, and entitled “Virtual White Lines for Delimiting PlannedExcavation Sites;”

U.S. publication no. 2009-0238415-A1, published Sep. 24, 2009, filedSep. 26, 2008, and entitled “Virtual White Lines for Delimiting PlannedExcavation Sites;”

U.S. publication no. 2009-0241046-A1, published Sep. 24, 2009, filedJan. 16, 2009, and entitled “Virtual White Lines for Delimiting PlannedExcavation Sites;”

U.S. publication no. 2009-0238416-A1, published Sep. 24, 2009, filedJan. 16, 2009, and entitled “Virtual White Lines for Delimiting PlannedExcavation Sites;”

U.S. publication no. 2009-0237408-A1, published Sep. 24, 2009, filedJan. 16, 2009, and entitled “Virtual White Lines for Delimiting PlannedExcavation Sites;”

U.S. publication no. 2011-0135163-A1, published Jun. 9, 2011, filed Feb.16, 2011, and entitled “Methods and Apparatus for Providing UnbufferedDig Area Indicators on Aerial Images to Delimit Planned ExcavationSites;”

U.S. publication no. 2009-0202101-A1, published Aug. 13, 2009, filedFeb. 12, 2008, and entitled “Electronic Manifest of Underground FacilityLocate Marks;”

U.S. publication no. 2009-0202110-A1, published Aug. 13, 2009, filedSep. 11, 2008, and entitled “Electronic Manifest of Underground FacilityLocate Marks;”

U.S. publication no. 2009-0201311-A1, published Aug. 13, 2009, filedJan. 30, 2009, and entitled “Electronic Manifest of Underground FacilityLocate Marks;”

U.S. publication no. 2009-0202111-A1, published Aug. 13, 2009, filedJan. 30, 2009, and entitled “Electronic Manifest of Underground FacilityLocate Marks;”

U.S. publication no. 2009-0204625-A1, published Aug. 13, 2009, filedFeb. 5, 2009, and entitled “Electronic Manifest of Underground FacilityLocate Operation;”

U.S. publication no. 2009-0204466-A1, published Aug. 13, 2009, filedSep. 4, 2008, and entitled “Ticket Approval System For and Method ofPerforming Quality Control In Field Service Applications;”

U.S. publication no. 2009-0207019-A1, published Aug. 20, 2009, filedApr. 30, 2009, and entitled “Ticket Approval System For and Method ofPerforming Quality Control In Field Service Applications;”

U.S. publication no. 2009-0210284-A1, published Aug. 20, 2009, filedApr. 30, 2009, and entitled “Ticket Approval System For and Method ofPerforming Quality Control In Field Service Applications;”

U.S. publication no. 2009-0210297-A1, published Aug. 20, 2009, filedApr. 30, 2009, and entitled “Ticket Approval System For and Method ofPerforming Quality Control In Field Service Applications;”

U.S. publication no. 2009-0210298-A1, published Aug. 20, 2009, filedApr. 30, 2009, and entitled “Ticket Approval System For and Method ofPerforming Quality Control In Field Service Applications;”

U.S. publication no. 2009-0210285-A1, published Aug. 20, 2009, filedApr. 30, 2009, and entitled “Ticket Approval System For and Method ofPerforming Quality Control In Field Service Applications;”

U.S. publication no. 2009-0324815-A1, published Dec. 31, 2009, filedApr. 24, 2009, and entitled “Marking Apparatus and Marking Methods UsingMarking Dispenser with Machine-Readable ID Mechanism;”

U.S. publication no. 2010-0006667-A1, published Jan. 14, 2010, filedApr. 24, 2009, and entitled, “Marker Detection Mechanisms for use inMarking Devices And Methods of Using Same;”

U.S. publication no. 2010-0085694 A1, published Apr. 8, 2010, filed Sep.30, 2009, and entitled, “Marking Device Docking Stations and Methods ofUsing Same;”

U.S. publication no. 2010-0085701 A1, published Apr. 8, 2010, filed Sep.30, 2009, and entitled, “Marking Device Docking Stations Having SecurityFeatures and Methods of Using Same;”

U.S. publication no. 2010-0084532 A1, published Apr. 8, 2010, filed Sep.30, 2009, and entitled, “Marking Device Docking Stations HavingMechanical Docking and Methods of Using Same;”

U.S. publication no. 2010-0088032-A1, published Apr. 8, 2010, filed Sep.29, 2009, and entitled, “Methods, Apparatus and Systems for GeneratingElectronic Records of Locate And Marking Operations, and Combined Locateand Marking Apparatus for Same;”

U.S. publication no. 2010-0117654 A1, published May 13, 2010, filed Dec.30, 2009, and entitled, “Methods and Apparatus for Displaying anElectronic Rendering of a Locate and/or Marking Operation Using DisplayLayers;”

U.S. publication no. 2010-0086677 A1, published Apr. 8, 2010, filed Aug.11, 2009, and entitled, “Methods and Apparatus for Generating anElectronic Record of a Marking Operation Including Service-RelatedInformation and Ticket Information;”

U.S. publication no. 2010-0086671 A1, published Apr. 8, 2010, filed Nov.20, 2009, and entitled, “Methods and Apparatus for Generating anElectronic Record of A Marking Operation Including Service-RelatedInformation and Ticket Information;”

U.S. publication no. 2010-0085376 A1, published Apr. 8, 2010, filed Oct.28, 2009, and entitled, “Methods and Apparatus for Displaying anElectronic Rendering of a Marking Operation Based on an ElectronicRecord of Marking Information;”

U.S. publication no. 2010-0088164-A1, published Apr. 8, 2010, filed Sep.30, 2009, and entitled, “Methods and Apparatus for Analyzing Locate andMarking Operations with Respect to Facilities Maps;”

U.S. publication no. 2010-0088134 A1, published Apr. 8, 2010, filed Oct.1, 2009, and entitled, “Methods and Apparatus for Analyzing Locate andMarking Operations with Respect to Historical Information;”

U.S. publication no. 2010-0088031 A1, published Apr. 8, 2010, filed Sep.28, 2009, and entitled, “Methods and Apparatus for Generating anElectronic Record of Environmental Landmarks Based on Marking DeviceActuations;”

U.S. publication no. 2010-0188407 A1, published Jul. 29, 2010, filedFeb. 5, 2010, and entitled “Methods and Apparatus for Displaying andProcessing Facilities Map Information and/or Other Image Information ona Marking Device;”

U.S. publication no. 2010-0198663 A1, published Aug. 5, 2010, filed Feb.5, 2010, and entitled “Methods and Apparatus for Overlaying ElectronicMarking Information on Facilities Map Information and/or Other ImageInformation Displayed on a Marking Device;”

U.S. publication no. 2010-0188215 A1, published Jul. 29, 2010, filedFeb. 5, 2010, and entitled “Methods and Apparatus for Generating Alertson a Marking Device, Based on Comparing Electronic Marking Informationto Facilities Map Information and/or Other Image Information;”

U.S. publication no. 2010-0188088 A1, published Jul. 29, 2010, filedFeb. 5, 2010, and entitled “Methods and Apparatus for Displaying andProcessing Facilities Map Information and/or Other Image Information ona Locate Device;”

U.S. publication no. 2010-0189312 A1, published Jul. 29, 2010, filedFeb. 5, 2010, and entitled “Methods and Apparatus for OverlayingElectronic Locate Information on Facilities Map Information and/or OtherImage Information Displayed on a Locate Device;”

U.S. publication no. 2010-0188216 A1, published Jul. 29, 2010, filedFeb. 5, 2010, and entitled “Methods and Apparatus for Generating Alertson a Locate Device, Based ON Comparing Electronic Locate Information TOFacilities Map Information and/or Other Image Information;”

U.S. publication no. 2010-0189887 A1, published Jul. 29, 2010, filedFeb. 11, 2010, and entitled “Marking Apparatus Having Enhanced Featuresfor Underground Facility Marking Operations, and Associated Methods andSystems;”

U.S. publication no. 2010-0256825-A1, published Oct. 7, 2010, filed Jun.9, 2010, and entitled “Marking Apparatus Having Operational Sensors ForUnderground Facility Marking Operations, And Associated Methods AndSystems;”

U.S. publication no. 2010-0255182-A1, published Oct. 7, 2010, filed Jun.9, 2010, and entitled “Marking Apparatus Having Operational Sensors ForUnderground Facility Marking Operations, And Associated Methods AndSystems;”

U.S. publication no. 2010-0245086-A1, published Sep. 30, 2010, filedJun. 9, 2010, and entitled “Marking Apparatus Configured To DetectOut-Of-Tolerance Conditions In Connection With Underground FacilityMarking Operations, And Associated Methods And Systems;”

U.S. publication no. 2010-0247754-A1, published Sep. 30, 2010, filedJun. 9, 2010, and entitled “Methods and Apparatus For Dispensing MarkingMaterial In Connection With Underground Facility Marking OperationsBased on Environmental Information and/or Operational Information;”

U.S. publication no. 2010-0262470-A1, published Oct. 14, 2010, filedJun. 9, 2010, and entitled “Methods, Apparatus, and Systems ForAnalyzing Use of a Marking Device By a Technician To Perform AnUnderground Facility Marking Operation;”

U.S. publication no. 2010-0263591-A1, published Oct. 21, 2010, filedJun. 9, 2010, and entitled “Marking Apparatus Having EnvironmentalSensors and Operations Sensors for Underground Facility MarkingOperations, and Associated Methods and Systems;”

U.S. publication no. 2010-0188245 A1, published Jul. 29, 2010, filedFeb. 11, 2010, and entitled “Locate Apparatus Having Enhanced Featuresfor Underground Facility Locate Operations, and Associated Methods andSystems;”

U.S. publication no. 2010-0253511-A1, published Oct. 7, 2010, filed Jun.18, 2010, and entitled “Locate Apparatus Configured to DetectOut-of-Tolerance Conditions in Connection with Underground FacilityLocate Operations, and Associated Methods and Systems;”

U.S. publication no. 2010-0257029-A1, published Oct. 7, 2010, filed Jun.18, 2010, and entitled “Methods, Apparatus, and Systems For AnalyzingUse of a Locate Device By a Technician to Perform an UndergroundFacility Locate Operation;”

U.S. publication no. 2010-0253513-A1, published Oct. 7, 2010, filed Jun.18, 2010, and entitled “Locate Transmitter Having Enhanced Features ForUnderground Facility Locate Operations, and Associated Methods andSystems;”

U.S. publication no. 2010-0253514-A1, published Oct. 7, 2010, filed Jun.18, 2010, and entitled “Locate Transmitter Configured to DetectOut-of-Tolerance Conditions In Connection With Underground FacilityLocate Operations, and Associated Methods and Systems;”

U.S. publication no. 2010-0256912-A1, published Oct. 7, 2010, filed Jun.18, 2010, and entitled “Locate Apparatus for Receiving EnvironmentalInformation Regarding Underground Facility Marking Operations, andAssociated Methods and Systems;”

U.S. publication no. 2009-0204238-A1, published Aug. 13, 2009, filedFeb. 2, 2009, and entitled “Electronically Controlled Marking Apparatusand Methods;”

U.S. publication no. 2009-0208642-A1, published Aug. 20, 2009, filedFeb. 2, 2009, and entitled “Marking Apparatus and Methods For Creatingan Electronic Record of Marking Operations;”

U.S. publication no. 2009-0210098-A1, published Aug. 20, 2009, filedFeb. 2, 2009, and entitled “Marking Apparatus and Methods For Creatingan Electronic Record of Marking Apparatus Operations;”

U.S. publication no. 2009-0201178-A1, published Aug. 13, 2009, filedFeb. 2, 2009, and entitled “Methods For Evaluating Operation of MarkingApparatus;”

U.S. publication no. 2009-0238417-A1, published Sep. 24, 2009, filedFeb. 6, 2009, and entitled “Virtual White Lines for Indicating PlannedExcavation Sites on Electronic Images;”

U.S. publication no. 2010-0205264-A1, published Aug. 12, 2010, filedFeb. 10, 2010, and entitled “Methods, Apparatus, and Systems forExchanging Information Between Excavators and Other Entities Associatedwith Underground Facility Locate and Marking Operations;”

U.S. publication no. 2010-0205031-A1, published Aug. 12, 2010, filedFeb. 10, 2010, and entitled “Methods, Apparatus, and Systems forExchanging Information Between Excavators and Other Entities Associatedwith Underground Facility Locate and Marking Operations;”

U.S. publication no. 2010-0259381-A1, published Oct. 14, 2010, filedJun. 28, 2010, and entitled “Methods, Apparatus and Systems forNotifying Excavators and Other Entities of the Status of in-ProgressUnderground Facility Locate and Marking Operations;”

U.S. publication no. 2010-0262670-A1, published Oct. 14, 2010, filedJun. 28, 2010, and entitled “Methods, Apparatus and Systems forCommunicating Information Relating to the Performance of UndergroundFacility Locate and Marking Operations to Excavators and OtherEntities;”

U.S. publication no. 2010-0259414-A1, published Oct. 14, 2010, filedJun. 28, 2010, and entitled “Methods, Apparatus And Systems ForSubmitting Virtual White Line Drawings And Managing Notifications InConnection With Underground Facility Locate And Marking Operations;”

U.S. publication no. 2010-0268786-A1, published Oct. 21, 2010, filedJun. 28, 2010, and entitled “Methods, Apparatus and Systems forRequesting Underground Facility Locate and Marking Operations andManaging Associated Notifications;”

U.S. publication no. 2010-0201706-A1, published Aug. 12, 2010, filedJun. 1, 2009, and entitled “Virtual White Lines (VWL) for DelimitingPlanned Excavation Sites of Staged Excavation Projects;”

U.S. publication no. 2010-0205555-A1, published Aug. 12, 2010, filedJun. 1, 2009, and entitled “Virtual White Lines (VWL) for DelimitingPlanned Excavation Sites of Staged Excavation Projects;”

U.S. publication no. 2010-0205195-A1, published Aug. 12, 2010, filedJun. 1, 2009, and entitled “Methods and Apparatus for Associating aVirtual White Line (VWL) Image with Corresponding Ticket Information foran Excavation Project;”

U.S. publication no. 2010-0205536-A1, published Aug. 12, 2010, filedJun. 1, 2009, and entitled “Methods and Apparatus for Controlling Accessto a Virtual White Line (VWL) Image for an Excavation Project;”

U.S. publication no. 2010-0228588-A1, published Sep. 9, 2010, filed Feb.11, 2010, and entitled “Management System, and Associated Methods andApparatus, for Providing Improved Visibility, Quality Control and AuditCapability for Underground Facility Locate and/or Marking Operations;”

U.S. publication no. 2010-0324967-A1, published Dec. 23, 2010, filedJul. 9, 2010, and entitled “Management System, and Associated Methodsand Apparatus, for Dispatching Tickets, Receiving Field Information, andPerforming A Quality Assessment for Underground Facility Locate and/orMarking Operations;”

U.S. publication no. 2010-0318401-A1, published Dec. 16, 2010, filedJul. 9, 2010, and entitled “Methods and Apparatus for Performing Locateand/or Marking Operations with Improved Visibility, Quality Control andAudit Capability;”

U.S. publication no. 2010-0318402-A1, published Dec. 16, 2010, filedJul. 9, 2010, and entitled “Methods and Apparatus for Managing Locateand/or Marking Operations;”

U.S. publication no. 2010-0318465-A1, published Dec. 16, 2010, filedJul. 9, 2010, and entitled “Systems and Methods for Managing Access toInformation Relating to Locate and/or Marking Operations;”

U.S. publication no. 2010-0201690-A1, published Aug. 12, 2010, filedApr. 13, 2009, and entitled “Virtual White Lines (VWL) Application forIndicating a Planned Excavation or Locate Path;”

U.S. publication no. 2010-0205554-A1, published Aug. 12, 2010, filedApr. 13, 2009, and entitled “Virtual White Lines (VWL) Application forIndicating an Area of Planned Excavation;”

U.S. publication no. 2009-0202112-A1, published Aug. 13, 2009, filedFeb. 11, 2009, and entitled “Searchable Electronic Records ofUnderground Facility Locate Marking Operations;”

U.S. publication no. 2009-0204614-A1, published Aug. 13, 2009, filedFeb. 11, 2009, and entitled “Searchable Electronic Records ofUnderground Facility Locate Marking Operations;”

U.S. publication no. 2011-0060496-A1, published Mar. 10, 2011, filedAug. 10, 2010, and entitled “Systems and Methods for Complex EventProcessing of Vehicle Information and Image Information Relating to aVehicle.;”

U.S. publication no. 2011-0093162-A1, published Apr. 21, 2011, filedDec. 28, 2010, and entitled “Systems And Methods For Complex EventProcessing Of Vehicle-Related Information;”

U.S. publication no. 2011-0093306-A1, published Apr. 21, 2011, filedDec. 28, 2010, and entitled “Fleet Management Systems And Methods ForComplex Event Processing Of Vehicle-Related Information Via Local AndRemote Complex Event Processing Engines;”

U.S. publication no. 2011-0093304-A1, published Apr. 21, 2011, filedDec. 29, 2010, and entitled “Systems And Methods For Complex EventProcessing Based On A Hierarchical Arrangement Of Complex EventProcessing Engines;”

U.S. publication no. 2010-0257477-A1, published Oct. 7, 2010, filed Apr.2, 2010, and entitled “Methods, Apparatus, and Systems for Documentingand Reporting Events Via Time-Elapsed Geo-Referenced ElectronicDrawings;”

U.S. publication no. 2010-0256981-A1, published Oct. 7, 2010, filed Apr.2, 2010, and entitled “Methods, Apparatus, and Systems for Documentingand Reporting Events Via Time-Elapsed Geo-Referenced ElectronicDrawings;”

U.S. publication no. 2010-0205032-A1, published Aug. 12, 2010, filedFeb. 11, 2010, and entitled “Marking Apparatus Equipped with TicketProcessing Software for Facilitating Marking Operations, and AssociatedMethods;”

U.S. publication no. 2011-0035251-A1, published Feb. 10, 2011, filedJul. 15, 2010, and entitled “Methods, Apparatus, and Systems forFacilitating and/or Verifying Locate and/or Marking Operations;”

U.S. publication no. 2011-0035328-A1, published Feb. 10, 2011, filedJul. 15, 2010, and entitled “Methods, Apparatus, and Systems forGenerating Technician Checklists for Locate and/or Marking Operations;”

U.S. publication no. 2011-0035252-A1, published Feb. 10, 2011, filedJul. 15, 2010, and entitled “Methods, Apparatus, and Systems forProcessing Technician Checklists for Locate and/or Marking Operations;”

U.S. publication no. 2011-0035324-A1, published Feb. 10, 2011, filedJul. 15, 2010, and entitled “Methods, Apparatus, and Systems forGenerating Technician Workflows for Locate and/or Marking Operations;”

U.S. publication no. 2011-0035245-A1, published Feb. 10, 2011, filedJul. 15, 2010, and entitled “Methods, Apparatus, and Systems forProcessing Technician Workflows for Locate and/or Marking Operations;”

U.S. publication no. 2011-0035260-A1, published Feb. 10, 2011, filedJul. 15, 2010, and entitled “Methods, Apparatus, and Systems for QualityAssessment of Locate and/or Marking Operations Based on Process Guides;”

U.S. publication no. 2010-0256863-A1, published Oct. 7, 2010, filed Apr.2, 2010, and entitled “Methods, Apparatus, and Systems for Acquiring andAnalyzing Vehicle Data and Generating an Electronic Representation ofVehicle Operations;”

U.S. publication no. 2011-0022433-A1, published Jan. 27, 2011, filedJun. 24, 2010, and entitled “Methods and Apparatus for Assessing LocateRequest Tickets;”

U.S. publication no. 2011-0040589-A1, published Feb. 17, 2011, filedJul. 21, 2010, and entitled “Methods and Apparatus for AssessingComplexity of Locate Request Tickets;”

U.S. publication no. 2011-0046993-A1, published Feb. 24, 2011, filedJul. 21, 2010, and entitled “Methods and Apparatus for Assessing RisksAssociated with Locate Request Tickets;”

U.S. publication no. 2011-0046994-A1, published Feb. 17, 2011, filedJul. 21, 2010, and entitled “Methods and Apparatus for Multi-StageAssessment of Locate Request Tickets;”

U.S. publication no. 2011-0040590-A1, published Feb. 17, 2011, filedJul. 21, 2010, and entitled “Methods and Apparatus for Improving aTicket Assessment System;”

U.S. publication no. 2011-0020776-A1, published Jan. 27, 2011, filedJun. 25, 2010, and entitled “Locating Equipment for and Methods ofSimulating Locate Operations for Training and/or Skills Evaluation;”

U.S. publication no. 2010-0285211-A1, published Nov. 11, 2010, filedApr. 21, 2010, and entitled “Method Of Using Coded Marking Patterns InUnderground Facilities Locate Operations;”

U.S. publication no. 2011-0137769-A1, published Jun. 9, 2011, filed Nov.5, 2010, and entitled “Method Of Using Coded Marking Patterns InUnderground Facilities Locate Operations;”

U.S. publication no. 2009-0327024-A1, published Dec. 31, 2009, filedJun. 26, 2009, and entitled “Methods and Apparatus for QualityAssessment of a Field Service Operation;”

U.S. publication no. 2010-0010862-A1, published Jan. 14, 2010, filedAug. 7, 2009, and entitled, “Methods and Apparatus for QualityAssessment of a Field Service Operation Based on GeographicInformation;”

U.S. publication No. 2010-0010863-A1, published Jan. 14, 2010, filedAug. 7, 2009, and entitled, “Methods and Apparatus for QualityAssessment of a Field Service Operation Based on Multiple ScoringCategories;”

U.S. publication no. 2010-0010882-A1, published Jan. 14, 2010, filedAug. 7, 2009, and entitled, “Methods and Apparatus for QualityAssessment of a Field Service Operation Based on Dynamic AssessmentParameters;”

U.S. publication no. 2010-0010883-A1, published Jan. 14, 2010, filedAug. 7, 2009, and entitled, “Methods and Apparatus for QualityAssessment of a Field Service Operation Based on Multiple QualityAssessment Criteria;”

U.S. publication no. 2011-0007076-A1, published Jan. 13, 2011, filedJul. 7, 2010, and entitled, “Methods, Apparatus and Systems forGenerating Searchable Electronic Records of Underground Facility Locateand/or Marking Operations;”

U.S. publication no. 2011-0131081-A1, published Jun. 2, 2011, filed Oct.29, 2010, and entitled “Methods, Apparatus, and Systems for Providing anEnhanced Positive Response in Underground Facility Locate and MarkingOperations;”

U.S. publication no. 2011-0060549-A1, published Mar. 10, 2011, filedAug. 13, 2010, and entitled, “Methods and Apparatus for AssessingMarking Operations Based on Acceleration Information;”

U.S. publication no. 2011-0117272-A1, published May 19, 2011, filed Aug.19, 2010, and entitled, “Marking Device with Transmitter forTriangulating Location During Locate Operations;”

U.S. publication no. 2011-0045175-A1, published Feb. 24, 2011, filed May25, 2010, and entitled, “Methods and Marking Devices with Mechanisms forIndicating and/or Detecting Marking Material Color;”

U.S. publication no. 2010-0088135 A1, published Apr. 8, 2010, filed Oct.1, 2009, and entitled, “Methods and Apparatus for Analyzing Locate andMarking Operations with Respect to Environmental Landmarks;”

U.S. publication no. 2010-0085185 A1, published Apr. 8, 2010, filed Sep.30, 2009, and entitled, “Methods and Apparatus for Generating ElectronicRecords of Locate Operations;”

U.S. publication no. 2011-0095885 A9 (Corrected Publication), publishedApr. 28, 2011, and entitled, “Methods And Apparatus For GeneratingElectronic Records Of Locate Operations;”

U.S. publication no. 2010-0090700-A1, published Apr. 15, 2010, filedOct. 30, 2009, and entitled “Methods and Apparatus for Displaying anElectronic Rendering of a Locate Operation Based on an Electronic Recordof Locate Information;”

U.S. publication no. 2010-0085054 A1, published Apr. 8, 2010, filed Sep.30, 2009, and entitled, “Systems and Methods for Generating ElectronicRecords of Locate And Marking Operations;” and

U.S. publication no. 2011-0046999-A1, published Feb. 24, 2011, filedAug. 4, 2010, and entitled, “Methods and Apparatus for Analyzing Locateand Marking Operations by Comparing Locate Information and MarkingInformation.”

It should be appreciated that all combinations of the foregoing conceptsand additional concepts discussed in greater detail below (provided suchconcepts are not mutually inconsistent) are contemplated as being partof the inventive subject matter disclosed herein. In particular, allcombinations of claimed subject matter appearing at the end of thisdisclosure are contemplated as being part of the inventive subjectmatter disclosed herein. It should also be appreciated that terminologyexplicitly employed herein that also may appear in any disclosureincorporated by reference should be accorded a meaning most consistentwith the particular concepts disclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings are not necessarily to scale, emphasis instead generallybeing placed upon illustrating the principles of the invention.

FIG. 1 shows an example in which a locate and marking operation isinitiated as a result of an excavator providing an excavation notice toa one-call center.

FIG. 2 illustrates one example of a conventional locate instrument setincluding a locate transmitter and a locate receiver.

FIGS. 3A and 3B illustrate a conventional marking device in an actuatedand non-actuated state, respectively.

FIG. 4 shows a perspective view of an example of an imaging-enabledmarking device that has a camera system and image analysis softwareinstalled therein for determining a type of surface being marked ortraversed, according to some embodiments of the present disclosure.

FIG. 5 illustrates a functional block diagram of an example of controlelectronics of an imaging-enabled marking device, according to someembodiments of the present disclosure.

FIG. 6 illustrates a functional block diagram of an example of referencedata that is stored locally at an imaging-enabled marking device,according to some embodiments of the present disclosure.

FIGS. 7A-F illustrate examples of reference histograms that representreference grayscale luminance distributions, which may be useful fordetermining a type of surface being marked or traversed, according tosome embodiments of the present disclosure.

FIG. 8 illustrates a functional block diagram of examples of inputdevices of an imaging-enabled marking device, according to someembodiments of the present disclosure.

FIG. 8A reproduces illustrative spectral signatures of three differentsurface types.

FIG. 9 illustrates a flow diagram of an example of a method of using acamera system and image analysis software for determining a type ofsurface being marked or traversed, according to some embodiments of thepresent disclosure.

FIG. 10 illustrates a functional block diagram of an example of a locateoperations system that includes one or more network of imaging-enabledmarking devices, according to some embodiments of the presentdisclosure.

FIG. 11 illustrates a flow diagram of an example of a method fordetermining a type of surface being marked or traversed, according tosome embodiments of the present disclosure.

DETAILED DESCRIPTION

Applicants have recognized and appreciated that collecting and analyzinginformation relating to a type of surface being marked (e.g., dirt,grass, sand, gravel, asphalt, concrete, etc.) may facilitate ensuringaccuracy of locate and marking operations. For example, Applicants haverecognized and appreciated that collecting and analyzing surface typeinformation may facilitate ensuring that an appropriate type of markingmaterial is applied. As a more specific example, some municipalitiesrequire that marking paint dispensed on streets and/or sidewalks fadeaway within a specified period of time (e.g., two to three weeks), so asto reduce any negative impact on the aesthetic appearance of the streetsand/or sidewalks. Therefore, it may be beneficial to detect whether thetype of surface being marked is pavement (e.g., asphalt or concrete, asopposed to dirt, grass, gravel, or sand) and, accordingly, select anappropriate formulation of marking material. As another example, somejurisdictions (e.g., federal, state, county, and/or municipality)require that locate marks remain recognizable for at least some periodof time (e.g., 10 to 14 days). Therefore, in some circumstances (e.g.,during summer or some other growing season), it may be beneficial todetect whether the type of surface being marked is grass and, if so, usea type of marking material (e.g., flags) other than paint. Such surfacetype detection may be performed at the beginning of a marking operation,and/or on an on-going basis throughout the marking operation. Forexample, if a surface type transition (e.g., from pavement to grass orvice versa) is detected, an alert may be generated to remind thetechnician to change to an appropriate type of marking material.

As another example, Applicants have recognized and appreciated thatcollecting and analyzing surface type information may facilitatedetecting undesirable operating conditions. For instance, if thehumidity of the operating environment is too great, marking materialsuch as paint may not adequately dry, or it may not remain in place onthe surface on which it is dispensed. Furthermore, acceptable ranges ofhumidity may differ depending on the type of surface being marked (e.g.,a humidity tolerance for grass may be lower than that for concrete ordirt). Therefore, detecting the type of surface being marked mayfacilitate determining whether current operating conditions (e.g.,humidity) are within acceptable limits.

Accordingly, systems, methods, and apparatus are provided herein forperforming surface type detection in connection with locate and markingoperations.

In some embodiments, one or more sensors may be employed to collectinformation regarding a surface, such as a ground surface on whichmarking material is to be dispensed to mark the presence or absence ofan underground facility. The collected sensor data may be analyzed toprovide an estimate of a type of the surface that is being sensed. Forinstance, based on the sensor data, it may be determined that thesurface being sensed is likely to be asphalt, concrete, wood, grass,dirt (or soil), brick, gravel, stone, snow, or any other surface type orcombination of surface types.

Various techniques may be used to process and analyze sensor data forpurposes of surface type detection. For instance, in some embodiments,the sensor data collected from a surface (or some representative dataderived the sensor data) may be compared against some previously storedreference data to identify one or more likely surface types for thesensed surface. As a more specific example, a surface signature may bederived from the sensor data and compared against a list of referencesignatures associated respectively with a list of different surfacetypes, so as to identify one or more candidate surface types whosereferences signatures mostly closely match the surface signature. Aconfidence score may be computed for each candidate surface type basedon the extent to which the surface signature matches the referencesignature corresponding to that candidate surface type.

As used herein, the term “signature” may be refer to any suitable set ofrepresentative data that can be used to identify a surface type for asensed surface. In some illustrative implementations, a signature maycontain part or all of the raw sensor data collected from a surface.Alternatively, or additionally, a signature may contain one or moreresults of transforming, filtering, augmenting, aggregating, and/orinterpreting the raw sensor data in any suitable manner. Althoughspecific examples of signatures are discussed in greater detail below,it should be appreciated that other types of signatures may also besuitable, depending on the sensing and analysis techniques that areemployed in each specific implementation.

Various types of sensors may be used collect information regarding asurface and may operate based on different physical principles. Forinstance, some sensors are designed to detect radiation in one or moreportions of the electromagnetic (EM) spectrum, whereas other sensors aredesigned to detect sound waves. Sensors based on other physicalprinciples may also be suitable, as aspects of the present disclosurerelating to sensing are not limited to any particular types of sensors.

As a specific example, a conventional still-image or video camera may beused as a sensor that detects visible light reflecting from a surface.Alternatively, various embodiments may use other image detectionhardware, including, but not limited to color-sensing chips, opticalflow chips, and the like. One or more images of the surface captured bythe camera may be analyzed using some suitable image analysis softwareto identify one or more characteristics (e.g., color, intensity,randomness, presence/absence of features such as lines, etc.) that maybe indicative of a surface type. An identified characteristic (e.g., thecolor “green”) may be used as a signature of the sensed surface and maybe compared against a list of reference signatures (e.g., “green” forgrass, “red” for brick, “black” for asphalt, etc.) to identify acandidate surface type for the sensed surface (e.g., grass).

As another example, one or more radiation sensors may be employed tomeasure an amount of electromagnetic radiation reflected by a surface ateach of one or more selected wavelengths or ranges of wavelengths (e.g.,visible light, infrared, ultraviolet, etc.). The source of the radiationmay be natural sun light and/or an artificial light source configured tooperate in conjunction with the radiation sensors (e.g., a calibratedlight source emitting light at a specific wavelength or range ofwavelengths, such as a broad spectrum IR light emitting diode). Thecollected sensor data (e.g., a percentage of radiation reflected by thesurface at each selected wavelength or range of wavelengths) may be usedas a spectral signature of the sensed surface and may be comparedagainst a list of reference spectral signatures correspondingrespectively to various surface types.

As yet another example, a thermal sensor may be employed to measure thetemperature of a surface by detecting infrared (IR) radiation from thesurface. As yet another example, a sonar sensor may be employed tomeasure sound waves reflected by the sensed surface. Illustrative usesof these sensors for purposes of surface type detection are discussed ingreater detail below, for example, in connection with FIGS. 8-9.

In some further embodiments, a combination of different sensing and/oranalysis techniques may be employed, which may lead to multiple surfacetype hypotheses for the sensed surface. These hypotheses may beaggregated and/or reconciled to further improve accuracy of surface typedetection. For example, a confidence score for a candidate surface typemay be increased if it is identified by two independent sensing and/oranalysis techniques as a likely match for the sensed surface. As anotherexample, a first matching surface type identified by a first sensingand/or analysis technique may be selected over a second matching surfacetype identified by a second sensing and/or analysis technique if theconfidence score assigned to the first matching surface type by thefirst sensing and/or analysis technique is higher than the confidencescore assigned to the second matching surface type by the second sensingand/or analysis technique. More generally, each candidate surface typemay be assigned a composite (or aggregate) confidence score based on theconfidence scores assigned to that candidate surface type underdifferent sensing and/or analysis techniques, and a candidate surfacetype having a highest composite confidence score may be identified as atop surface type hypothesis for the sensed surface. For instance, insome implementations, the composite confidence score may be a weightedsum of the component confidence scores, using weights associatedrespectively with the different sensing and/or analysis techniques.

In yet some further embodiments, some of the sensors used to collectdata from a surface may be attached to a marking device (e.g., themarking device 50 shown in FIGS. 3A-B), so that sensor data may becollected from the surface as it is being marked (or shortly before orafter it is marked). Each such sensor may be attached to the markingdevice externally (e.g., outside a housing of the marking device) orinternally (e.g., inside a housing of the marking device), and may becommunicatively coupled to one or more other components of the markingdevice that are configured to receive and process sensor data. Forexample, a sensor may be communicatively coupled to a processor and amemory of the marking device, so that sensor data can be stored in thememory and analyzed by the processor. Additionally, or alternatively,the sensor data may be transmitted via a communication interface of themarking device to another computer for storage and/or analysis.

Applicants have further recognized and appreciated that the output ofsurface type detection (e.g., one or more surface type hypotheses) maybe used by one or more other applications related to the management oflocate and marking operations. For example, in one implementation, asurface type detection output may be used by a workflow application toautomatically select and/or recommend an appropriate type of markingmaterial to be applied to the sensed surface. In another implementation,a surface type detection output may be used by a quality controlapplication to determine whether certain adverse operating conditionexists (e.g., whether the humidity is too high for applying paint, orwhether there is ice on the surface to be marked). The quality controlapplication may react in real time to the detection of an adversecondition, for example, by sending an alert to a technician performingthe locate and marking operation. Alternatively, or additionally, thequality control application may flag the incident as requiring furtherreview, and a supervisor may determine whether any corrective action(e.g., a re-mark operation) may be needed and/or whether the technicianshould receive additional training.

In some instances, the information collected during the markingoperation may also be examined by a regulator and/or an insurer forauditing purposes (e.g., to verify whether the locate and markingoperation has been proper conducted). As another example, the electronicrecord may be analyzed during damage investigation in the event of anaccident during subsequent excavation (e.g., as evidence that a certaintype of marking material was dispensed at a certain location).

Following below are more detailed descriptions of various conceptsrelated to, and embodiments of, inventive systems, methods and apparatusfor surface type detection in connection with locate and markingoperations. It should be appreciated that various concepts introducedabove and discussed in greater detail below may be implemented in any ofnumerous ways, as the disclosed concepts are not limited to anyparticular manner of implementation. Examples of specificimplementations and applications are provided primarily for illustrativepurposes.

In some illustrative embodiments, a marking device is provided that hasa camera system and image analysis software installed therein fordetermining a type of surface being marked or traversed (hereaftercalled imaging-enabled marking device). In alternative embodiments,image analysis software may be located elsewhere, such as a separatecomputer processing unit on the marking device, or a remote server incommunication with the marking device by wireless communicationtechnology. In still further embodiments the marking device also maycollect data onto a local storage medium for later analysis, which maylater be transferred to a computer system for processing, e.g., overUSB. Examples of types of surfaces that may be identified using theimaging-enabled marking device may include, but are not limited to,asphalt, concrete, wood, grass, dirt (or soil), brick, gravel, stone,snow, and the like. Additionally, some types of surfaces may be paintedor unpainted (e.g., painted concrete vs. unpainted concrete). More thanone type of surface may be present at a jobsite.

The image analysis software may include any one or more algorithms thatare useful for automatically determining a type of surface being markedor traversed. More specifically, image analysis software may execute oneor more distinct processes for incrementally determining and/orotherwise confirming a level of confidence of matching a certain surfacetype with a surface being marked or traversed. By way of example, theexecution of a certain algorithm may determine a certain level ofconfidence of the surface being unpainted concrete. The execution ofanother type of algorithm may confirm, validate, verify, and/orotherwise support the results of the first algorithm and, thereby,increase the level of confidence of the surface being unpaintedconcrete. The execution of yet another type of algorithm may confirm,validate, verify, and/or otherwise support the results of the first andsecond algorithms and, thereby, further increase the level of confidenceof the surface being unpainted concrete, and so on until a finalconfidence level of the surface type is determined.

Additionally, for each algorithm, once a level of confidence of matchingis determined, for example, in the form of a numerical confidence score,the image analysis software may be capable of dynamically setting aweight factor to be applied to the confidence score. A final confidencescore may be calculated based on the individual confidence scores outputby the one or more algorithms of image analysis software and theassociated weight factors.

In certain embodiments, the camera system may include one or moredigital video cameras. In one example, the process of automaticallydetermining a type of surface being marked or traversed may be based onsensing motion of the imaging-enabled marking device. That is, any timethat imaging-enabled marking device is in motion, at least one of thedigital video cameras may be activated and processing of image data mayoccur, such as the processing of pixel intensities and/or colorcoordinates.

In other embodiments, other devices may be used in combination with thecamera system. These other devices may include, but are not limited to,one or more the following types of sensors, configured to collect sensordata either independently or in one or more suitable combinations: sonarsensor, inertial measurement unit, infrared sensor, temperature sensor,light sensor, and digital audio recorder.

Referring to FIG. 4, a perspective view of an example of animaging-enabled marking device 100 that has a camera system and imageanalysis software installed therein for determining a type of surfacebeing marked or traversed is presented. More specifically, FIG. 4 showsan imaging-enabled marking device 100 that is an electronic markingdevice capable of determining a type of surface being marked ortraversed using a camera system and image analysis software that isinstalled therein.

In one example, the imaging-enabled marking device 100 may includecertain control electronics 110 and one or more digital video cameras112. The control electronics 110 may be used for managing the overalloperations of the imaging-enabled marking device 100. More details of anexample of the control electronics 110 are described with reference toFIG. 5.

The one or more digital video cameras 112 may be any standard digitalvideo cameras that have a frame rate and resolution that is suitable foruse in the imaging-enabled marking device 100. Each digital video camera112 may be a universal serial bus (USB) digital video camera. In oneexample, each digital video camera 112 may be the Sony PlayStation®Eyevideo camera that has a 10-inch focal length and is capable of capturing60 frames/second, where each frame is, for example, 640×480 pixels. Inthis example, a suitable placement of digital video camera 112 onimaging-enabled marking device 100 may be about 10 to 13 inches from thesurface to be marked or traversed, when the marking device 100 is heldby a technician during normal use. Certain frames of the image data(e.g., every n^(th) frame) from digital video camera 112 may be storedin any standard or proprietary image file format (e.g., JPEG, BMP, TIFF,etc.).

Information from more than one digital video camera 112 may be useful toimage analysis software 114 for providing more image data to processwhen determining a type of surface being marked or traversed byimaging-enabled marking device 100. By way of example, imaging-enabledmarking device 100 may include a configuration of two digital videocameras 112. With respect to the body of imaging-enabled marking device100, the two digital video cameras 112 may be mounted in any usefulconfiguration, such as side-by-side, one behind the other, in the sameplane, not in the same plane, or any combinations thereof. Preferably,the fields of view (FOV) of both digital video cameras 112 have someamount of overlap, regardless of the mounting configuration.

Certain image analysis software 114 may reside at and execute on thecontrol electronics 110 of the imaging-enabled marking device 100. Theimage analysis software 114 may be any suitable image analysis softwarefor processing digital video output (e.g., from at least one digitalvideo camera 112). In order to conserve processing resources of thecontrol electronics 110, the image analysis software 114 may performimage analysis processes on, for example, every n^(th) frame (e.g.,every 5^(th), 10^(th) or 20^(th) frame) of the image data from thedigital video camera 112. The image analysis software 114 may include,for example, one or more algorithms for performing any useful imageanalysis processes with respect to determining a type of surface beingmarked or traversed from digital video that is captured using thedigital video camera 112. More details of examples of algorithms thatmay be implemented in the image analysis software 114 are described withreference to FIG. 5.

The imaging-enabled marking device 100 may include one or more devicesfor use in combination with the digital video cameras 112 and the imageanalysis software 114. For example, certain input devices 116 may beintegrated into or otherwise connected to the control electronics 110.Input devices 116 may be, for example, any systems, sensors, and/ordevices that are useful for acquiring and/or generating data that may beused in combination with the digital video cameras 112 and the imageanalysis software 114 for determining a type of surface being marked ortraversed, according to the present disclosure. More details of examplesof input devices 116 are described with reference to FIG. 8.

The components of the imaging-enabled marking device 100 may be poweredby a power source 118. The power source 118 may be any power source thatis suitable for use in a portable device, such as, but not limited to,one or more rechargeable batteries, one or more non-rechargeablebatteries, a solar electrovoltaic panel, a standard AC power plugfeeding an AC-to-DC converter, and the like.

FIG. 4 also shows that the imaging-enabled marking device 100 mayinclude a light source 120. In one example, the light source 120 is awhite light source that is powered by the power source 118. The lightsource 120 may be used for illuminating the surface to be marked ortraversed.

Referring to FIG. 5, a functional block diagram of an example of thecontrol electronics 110 of the imaging-enabled marking device 100 of thepresent disclosure is presented. In this example, control electronics110 may include, but is not limited to, the image analysis software 114shown in FIG. 4, a processing unit 122, a quantity of local memory 124,a communication interface 126, a user interface 128, a location trackingsystem 130, and an actuation system 132.

The image analysis software 114 may be executed by the processing unit122. The processing unit 122 may be any general-purpose processor,controller, or microcontroller device that is capable of managing theoverall operations of the imaging-enabled marking device 100, includingmanaging data that is returned from any component thereof. The localmemory 124 may be any volatile or non-volatile data storage device, suchas, but not limited to, a random access memory (RAM) device or aremovable memory device (e.g., a USB flash drive).

The communication interface 126 may be any wired and/or wirelesscommunication interface for connecting to a network (e.g., a local areanetwork such as an enterprise intranet, a wide area network, or theInternet) and by which information (e.g., the contents of the localmemory 124) may be exchanged with other devices connected to thenetwork. Examples of wired communication interfaces may be implementedaccording to various interface protocols, including, but not limited to,USB protocols, RS232 protocol, RS422 protocol, IEEE 1394 protocol,Ethernet protocols, optical protocols (e.g., relating to communicationsover fiber optics), and any combinations thereof. Examples of wirelesscommunication interfaces may be implemented according to variouswireless technologies, including, but not limited to, Bluetooth®,ZigBee®, Wi-Fi/IEEE 802.11, Wi-Max, various cellular protocols, InfraredData Association (IrDA) compatible protocols, Shared Wireless AccessProtocol (SWAP), and any combinations thereof.

The user interface 128 may be any mechanism or combination of mechanismsby which a user may operate the imaging-enabled marking device 100 andby which information that is generated by the imaging-enabled markingdevice 100 may be presented to the user. For example, the user interface128 may include, but is not limited to, a display, a touch screen, oneor more manual pushbuttons, one or more light-emitting diode (LED)indicators, one or more toggle switches, a keypad, an audio output(e.g., speaker, buzzer, or alarm), a wearable interface (e.g., dataglove), and any combinations thereof.

The location tracking system 130 may include any device that candetermine its geographical location to a certain degree of accuracy. Forexample, the location tracking system 130 may include a globalpositioning system (GPS) receiver, such as a global navigation satellitesystem (GNSS) receiver. A GPS receiver may provide, for example, anystandard format data stream, such as a National Marine ElectronicsAssociation (NMEA) data stream. The location tracking system 130 mayalso include an error correction component (not shown), which may be anymechanism for improving the accuracy of the geo-location data.

The actuation system 132 may include a mechanical and/or electricalactuator mechanism (not shown) that may be coupled to an actuator thatcauses the marking material to be dispensed from the marking dispenserof the imaging-enabled marking device 100. Actuation means starting orcausing the imaging-enabled marking device 100 to work, operate, and/orfunction. Examples of actuation may include, but are not limited to, anylocal or remote, physical, audible, inaudible, visual, non-visual,electronic, electromechanical, biomechanical, biosensing or othersignal, instruction, or event. Actuations of the imaging-enabled markingdevice 100 may be performed for any purpose, such as, but not limitedto, dispensing marking material and capturing any information of anycomponent of the imaging-enabled marking device 100 without dispensingmarking material. In one example, an actuation may occur by pulling orpressing a physical trigger of the imaging-enabled marking device 100that causes the marking material to be dispensed.

FIG. 5 also shows one or more digital video cameras 112 connected to thecontrol electronics 110 of the imaging-enabled marking device 100. Inparticular, image data 134 from the digital video cameras 112 may bepassed (e.g., frame by frame) to the processing unit 122 and processedby the image analysis software 114. In one example, every n^(th) frame(e.g., every 5^(th), 10^(th) or 20^(th) frame) of the image data 134 maybe processed and stored in the local memory 124. In this way, theprocessing capability of the processing unit 122 may be optimized. FIG.5 shows that the image analysis software 114 may include one or morealgorithms, which may be any task-specific algorithms with respect toprocessing the digital video output of digital video cameras 112 fordetermining a type of surface being marked or traversed. The results ofexecuting the operations of the image analysis software 114 may becompiled into surface type data 136, which may also be stored in thelocal memory 124.

Examples of these task-specific algorithms that may be part of the imageanalysis software 114 include, but are not limited to, a motiondetection algorithm 138, a pixel value analysis algorithm 140, a coloranalysis algorithm 142, a pixel entropy algorithm 144, an edge detectionalgorithm 146, a line detection algorithm 148, a boundary detectionalgorithm 150, a compression analysis algorithm 152, a surface historyalgorithm 154, and a dynamically weighted confidence level algorithm156. One reason for executing multiple algorithms in the process ofdetermining a type of surface being marked or traversed may be that anygiven single algorithm may be more or less effective for determiningcertain types of surfaces. Therefore, the collective output of multiplealgorithms is useful for making a final determination of a type ofsurface being marked or traversed, which is further described withreference to the method of FIG. 9.

Certain predetermined reference data 158 may be stored in the localmemory 124. The contents of the reference data 158 may be anyinformation that is useful to the image analysis software 114 and, inparticular, to the motion detection algorithm 138, the pixel valueanalysis algorithm 140, the color analysis algorithm 142, the pixelentropy algorithm 144, the edge detection algorithm 146, the linedetection algorithm 148, the boundary detection algorithm 150, thecompression analysis algorithm 152, the surface history algorithm 154,and the dynamically weighted confidence level algorithm 156, and anycombinations thereof. An example of the contents of the reference data158 is shown in FIG. 6.

Referring to FIG. 6, a functional block diagram of an example of thereference data 158 that is stored locally at the imaging-enabled markingdevice is presented. In this example, the reference data 158 includes,but is not limited to: reference histogram data 160, which is associatedwith the pixel value analysis algorithm 140; reference color data 162,which is associated with the color analysis algorithm 142; referenceentropy values 164, which is associated with the pixel entropy algorithm144; reference hue and saturation data 166, which is also associatedwith the color analysis algorithm 142; reference edge data 168, which isassociated with the edge detection algorithm 146; reference line data170, which is associated with the line detection algorithm 148;reference boundary data 172, which is associated with the boundarydetection algorithm 150; reference compression data 174, which isassociated with the compression analysis algorithm 152; and anycombinations thereof. Referring to FIGS. 5 and 6, the operation of thealgorithms of the image analysis software 114 and the associated thereference data 158 may be summarized as follows.

The motion detection algorithm 138 may be any algorithm for processinginformation from any mechanism that may be used for determining whetherthe imaging-enabled marking device 100 is in motion. In one example, themotion detection algorithm 138 may query readings from an inertialmeasurement unit (IMU), which is one example of an input device 116. TheIMU indicates, for example, the start and end of motion with respect tothe imaging-enabled marking device 100. More details of an example of anIMU are described with reference to FIG. 10. In another example, themotion detection algorithm 138 may query the output of an optical flowalgorithm (not shown) that is used to process the image data 134 from atleast one digital video camera 112 and determine whether imaging-enabledmarking device 100 is in motion. An optical flow algorithm performs anoptical flow calculation, which is well known, for determining thepattern of apparent motion of the source digital video camera 112 and,thereby, determines when the imaging-enabled marking device 100 is inmotion. In one example, the optical flow algorithm may be based on thePyramidal Lucas-Kanade method for performing the optical flowcalculation. In yet another example, the motion detection algorithm 138may use the combination of an IMU and the optical flow algorithm todetermine whether the imaging-enabled marking device 100 is in motion.

In one example, the digital video cameras 112 are activated only when itis sensed that the imaging-enabled marking device 100 is in motion. Thismode of operation may allow surface detection regardless of whethermarking material is being dispensed. In another example, only when themotion detection algorithm 138 indicates that the imaging-enabledmarking device 100 is in motion are the digital video cameras 112activated.

In another example, the digital video cameras 112 and associatedoperations of the image analysis software 114 may be actuation-based,i.e., based on the state of the actuation system 132. For example, eachtime the physical trigger of the imaging-enabled marking device 100 ispulled or pressed, the digital video cameras 112 and associatedoperations of the image analysis software 114 are activated. Thisalternative mode of operation may allow surface detection only whenmarking material is being dispensed.

In yet another example, the digital video cameras 112 and associatedoperations of image analysis software 114 may be started and stopped byany mechanisms, such as manually by the user or by programming. In yetanother example, once processes of image analysis software 114 areinitiated the software may be programmed to run for a certain amount oftime (e.g., a few seconds). In any case, once digital video camera 112is activated, image analysis software 114 may be programmed to processevery n^(th) frame (e.g., every 5^(th), 10^(th) or 20^(th) frame) ofimage data 134.

The pixel value analysis algorithm 140 may be used to generate agrayscale luminance distribution histogram of the current surface beingmarked or traversed. The pixel value analysis algorithm 140 may thencompare the current grayscale luminance distribution histogram to thereference histogram data 160 stored in the reference data 158.Applicants have recognized and appreciated that certain types ofsurfaces may have respective characteristic grayscale luminancedistributions. Accordingly, the reference histogram data 160 may includea set of characteristic grayscale luminance distributions for certaintypes of surfaces. The output of the pixel value analysis algorithm 140may include a confidence level of the current surface matching a certainsurface type, as described with reference to FIG. 9, wherein thisconfidence level is derived based on grayscale luminance distributions.

Referring to FIGS. 7A-F, examples of characteristic reference histogramsthat are stored in the reference histogram data 160 of the referencedata 158 are presented. The reference histograms in the reference data158 represent characteristic grayscale luminance distributions forvarious types of surfaces. For example, FIG. 7A shows an example of areference histogram 700A that is characteristic of the asphalt surfacetype. In this illustration, each line in the reference histogram 700Arepresents data collected from an image of some asphalt surface. Eachpixel in this image is associated with a value ranging from 1 to 10,based on the luminous intensity of the pixel. For each value i from 1 to10 along the horizontal axis, a point (i, y_(i)) is plotted, where y_(i)is the number of pixels in the image having luminous intensity value i.The line representing the image is then plotted by connecting the tenpoints (i, y_(i)), i=1, . . . , 10. Similarly, FIG. 7B shows an exampleof a reference histogram 700B that is characteristic of the mulchsurface type, FIG. 7C shows an example of a reference histogram 700Cthat is characteristic of the brick surface type, FIG. 7D shows anexample of a reference histogram 700D that is characteristic of thegrass surface type, FIG. 7E shows an example of a reference histogram700E that is characteristic of the painted concrete surface type, andFIG. 7F shows an example of a reference histogram 700F that ischaracteristic of the unpainted concrete surface type.

Although FIGS. 7A-F illustrate examples of surface types andcorresponding reference histograms, the reference histogram data 160 isnot limited to these specific examples. The reference histogram data 160may include any number of reference histograms corresponding to anynumber and types of surfaces.

While certain types of surfaces may have fairly distinguishablecharacteristic grayscale luminance distributions, other types of surfacemay have less distinguishable characteristic grayscale luminancedistributions. Accordingly, the pixel value analysis algorithm 140 maybe more or less effective for a given type of surfaces. Therefore, itmay be beneficial to run other image analysis processes in combinationwith the pixel value analysis algorithm 140 in order to confirm,validate, verify, and/or otherwise support any output of the pixel valueanalysis algorithm 140. For example, referring again to FIGS. 7A-F, thehistograms for asphalt (e.g., the reference histogram 700A), brick(e.g., the reference histogram 700C), painted concrete (e.g., thereference histogram 700E), and unpainted concrete (e.g., the referencehistogram 700F) may be fairly distinctive. By contrast, the histogramsfor mulch (e.g., reference histogram 700B) and grass (e.g., referencehistogram 700D) are not as distinctive and may be confused with eachother. Therefore, the pixel value analysis algorithm 140 may be moreeffective for determining asphalt, brick, painted concrete, andunpainted concrete, but less effective for determining mulch and grass.

Applicants have further appreciated that certain types of surfaces mayhave distinctly characteristic colors. Accordingly, the color analysisalgorithm 142 may be used to perform a color matching operation. Forexample, the color analysis algorithm 142 may be used to analyze the RGBcolor data (or color data in accordance with some suitable color modelother than the RGB model) of certain frames of the image data 134 fromthe digital video cameras 112. The color analysis algorithm 142 may thendetermine the most prevalent color that is present in the image frames.Next, the color analysis algorithm 142 may correlate the most prevalentcolor found in the image frames to a certain type of surface. Table 1below shows an example of the correlation of surface type to color. Thecontents of Table 1 may be stored, for example, in the reference colordata 162 of the reference data 158.

TABLE 1 Example correlation of surface type to color. Surface Type ColorAsphalt Black Mulch Medium Brown Brick Red Grass Green Unpaintedconcrete Gray Dirt Light Brown

The output of the color analysis algorithm 142 may include a confidencelevel of the current surface matching a certain surface type, asdescribed with reference to FIG. 9, wherein this confidence level isbased on analysis of the most prevalent color. Because the mostprevalent colors of certain types of surfaces may be similar (e.g.,concrete and asphalt, and mulch and brick), it may be beneficial to runother image analysis processes in combination with the color matchingoperation of the color analysis algorithm 142 in order to confirm,validate, verify, and/or otherwise support any output of the coloranalysis algorithm 142.

In an HSV (respectively, HSB or HSI) color coordinate system, colors canbe specified according to their hue, saturation, and value(respectively, brightness or intensity) components. Applicants havefurther recognized and appreciated that certain types of surfaces mayhave distinctly characteristic hue and saturation. Accordingly, thecolor analysis algorithm 142 may also be used to analyze the hue andsaturation aspects of the image data 134 from the digital video cameras112. Color analysis algorithm 142 may then correlate the hue andsaturation that is found in the image data 134 to a certain type ofsurface. The correlation of hue and saturation to surface type may bestored in the reference hue and saturation data 166 of reference data158.

The output of this operation of the color analysis algorithm 142 mayalso include a confidence level of the current surface matching acertain surface type, as described with reference to FIG. 9, whereinthis confidence level is based on analysis of hue and saturation.Because the hue and saturation characteristics of certain types ofsurfaces may be similar, it may be beneficial to run other imageanalysis processes in combination with the hue and saturation analysisof the color analysis algorithm 142 in order to confirm, validate,verify, and/or otherwise support any output of the color analysisalgorithm 142.

The pixel entropy algorithm 144 may be any suitable software algorithmfor measuring a degree of randomness of an image in the image data 134from the digital video cameras 112. Randomness of an image may be ameasure of, for example, the presence or absence of predictable patternsin the image. As a more specific example, the pixel entropy algorithm144 may compute an entropy value for an image in the image data 134based on the image's grayscale luminance distribution. Alternatively, oradditionally, the pixel entropy algorithm 144 may compute the entropyvalue based on the image's color distribution. Further still, the pixelentropy algorithm 144 may compute the entropy value based on a jointdistribution of luminance and color for the image. Thus, an image thatis more varied in color and/or intensity may have a higher entropy valuecompared to an image that is less varied in color and/or intensity.Table 2 below shows an example of the correlation of surface type toaverage entropy, where the average entropy for each surface type may becomputed based on entropy values of a sufficiently large number ofimages of that surface type. The contents of Table 2 may be stored, forexample, in the reference entropy values 164 of the reference data 158.

TABLE 2 Example correlation of surface type to average entropy. AverageSurface Type Entropy Value Asphalt 6.107 Mulch 7.517 Brick 6.642 Grass7.546 Painted concrete 4.675 Unpainted concrete 6.300

In operation, the pixel entropy algorithm 144 may determine an entropyvalue of a current image in the image data 134 and compares this valueto the values in the reference entropy values 164 (see Table 2). Theoutput of pixel entropy algorithm 144 may include a confidence level ofthe current surface matching a certain surface type, as described withreference to FIG. 9, wherein this confidence level is based on analysisof entropy values. Because average entropy values of certain types ofsurfaces may be similar (e.g., mulch and grass), it may be beneficial torun other image analysis processes in combination with the pixel entropyalgorithm 144 in order to confirm, validate, verify, and/or otherwisesupport any output of the pixel entropy algorithm 144.

Edge detection is a process of identifying points in a digital image atwhich image brightness changes sharply (e.g., a process of detectingextreme pixel differences). The edge detection algorithm 146 is used toperform edge detection on certain frames of the image data 134 from atleast one digital video camera 112. In one example, the edge detectionalgorithm 146 may use the Sobel operator, which is well known. The Sobeloperator calculates a gradient of image intensity at each point, givinga direction of largest increase from light to dark and/or from one colorto another and a rate of change in that direction. The result thereforeshows how “abruptly” or “smoothly” the image changes at that point and,therefore, how likely it is that that part of the image represents anedge, as well as how that edge is likely to be oriented.

The edge detection algorithm 146 may then correlate any edges found to acertain type of surface. For example, an image of a certain type ofsurface (e.g., mulch) may contain more edges per unit area compared toan image of another type of surface (e.g., painted concrete). Thecorrelation of the edge characteristics to surface type may be stored inthe reference edge data 168 of reference data 158. The output of theedge detection algorithm 146 may include a confidence level of thecurrent surface matching a certain surface type, as described withreference to FIG. 9, wherein this confidence level is based on analysisof edge characteristics. Because edge characteristics of certain typesof surfaces may be similar (e.g., mulch and grass), it may be beneficialto run other image analysis processes in combination with the edgedetection algorithm 146 in order to confirm, validate, verify, and/orotherwise support any output of the edge detection algorithm 146.

Additionally, one or more results of edge detection may be used by theline detection algorithm 148 for further processing to determine linecharacteristics of certain frames of the image data 134 from at leastone digital video camera 112. Like the edge detection algorithm 146, theline detection algorithm 148 may be based on one or more edge detectionprocesses that use, for example, the Sobel operator. The line detectionalgorithm 148 may group together edges output by the edge detectionprocesses based on the edges' locations, lengths, and/or orientations.For example, in one implementation, the line detection algorithm 148 mayoutput a detected line when the total length of a group of adjacentedges exceed a certain threshold (e.g., 10 pixels).

Applicants have appreciated and recognized that different surface typesmay contain different line patters. For example, on a brick surface,lines are present between bricks. As another example, on a sidewalksurface, lines are present between sections of concrete. Therefore, thecombination of edge detection algorithm 146 and line detection algorithm148 may be used for recognizing the presence of lines that are, forexample, repetitive, straight, and have corners. The line detectionalgorithm 148 may then correlate any lines found to a certain type ofsurface. The correlation of line characteristics to surface type may bestored in the reference line data 170 of reference data 158. The outputof the line detection algorithm 148 may include a confidence level ofthe current surface matching a certain surface type, as described withreference to FIG. 9, wherein this confidence level is based on analysisof line characteristics. Because line characteristics of certain typesof surfaces may be similar, it may be beneficial to run other imageanalysis processes in combination with the line detection algorithm 148in order to confirm, validate, verify, and/or otherwise support anyoutput of line detection algorithm 148.

Boundary detection is a process of detecting a boundary between two ormore surface types. The boundary detection algorithm 150 is used toperform boundary detection on certain frames of the image data 134 fromat least one digital video camera 112. In one example, the boundarydetection algorithm 150 first analyzes the four corners of a frame. Whentwo or more corners indicate different types of surfaces, the frame ofthe image data 134 may be classified as a “multi-surface” frame. Onceclassified as a “multi-surface” frame, it may be beneficial to run theedge detection algorithm 146 and/or the line detection algorithm 148,for example, to divide the frame into two or more subsections. Theboundary detection algorithm 150 may then analyze the two or moresubsections using any image analysis processes of the present disclosurefor determining a type of surface found in any of the two or moresubsections.

The output of boundary detection algorithm 150 may include a confidencelevel of each frame subsection matching a certain surface type, asdescribed with reference to FIG. 9. When two or more frame subsectionsindicate a high probability of different surface types, this mayindicate that the original frame is likely to contain a boundary betweentwo or more surface types. In one example, the boundary detectionalgorithm 150 may be executed only when a low degree of matching isreturned for all surface types for the original frame by other imageanalysis processes of the disclosure.

The compression analysis algorithm 152 may be any suitable softwarealgorithm for performing a compression operation on image data. As iswell known, in a compression operation such as standard JPEG, a discretecosine transform (DCT) may be applied to blocks of pixels to transformthe data into a frequency domain, thereby facilitating removing finedetails in the image (e.g., high frequency components) that are lessperceptible to humans. Applicants have recognized and appreciated thatimages of different surface types may be capable of differentcompression ratios when the same or similar compression routine isapplied. Accordingly, the compression analysis algorithm 152 may be usedto perform a compression routine, such as a standard JPEG compressionroutine using DCT, on frames of the image data 134 from the digitalvideo cameras 112. The output of the compression routine may include apercent compression value, which may be correlated to a certain type ofsurfaces. Table 3 shows an example of the correlation of surface type tocharacteristic compression ratio when applying a standard JPEGcompression routine. The contents of Table 3 are an example of theinformation stored in reference compression data 174.

TABLE 3 Example correlation of surface type to compression ratio. Rangeof Percent (%) Surface Type Compression Asphalt about 96.6 to 96.2 Dirtabout 95.5 to 95.1 Brick about 95.1 to 94.7 Grass about 93.4 to 93.0Unpainted concrete about 96.4 to 95.7 Painted concrete about 98.9 to98.1

Referring to Table 3, the example percent compression values areobtained under the following conditions: (1) all images being about thesame resolution; (2) all images being about the same color depth; (3)all images being about the same size in memory (e.g., about 1 megabyte);and (4) the “loss” variable being set to about 50%, 0% being compressedto almost no recognition and 100% being substantially lossless.

The output of the compression analysis algorithm 152 may include aconfidence level of the current surface matching a certain surface type,based on correlating the achieved compression ratio to a surface type,as described with reference to FIG. 9. Because compression ratios ofcertain types of surfaces may be similar, it may be beneficial to runother image analysis processes in combination with the compressionanalysis algorithm 152 in order to confirm, validate, verify, and/orotherwise support any output of the compression analysis algorithm 152.

The surface history algorithm 154 is a software algorithm for performinga comparison of a current surface type as determined by one or more ofthe aforementioned algorithms (either separately or in combination) tohistorical surface type information. In one example, the surface historyalgorithm 154 may compare a candidate surface type for a current frameof the image data 134 to surface type information of previous frames ofthe image data 134. For example, if there is a question of a currentsurface type being brick vs. wood, historical information of previousframes of the image data 134 may indicate that the surface type is brickand, therefore, it is most likely that the current surface type isbrick, not wood.

In some embodiments, an output (e.g., a confidence level of matching) ofeach algorithm of the present disclosure for determining a type ofsurface being marked or traversed (e.g., the pixel value analysisalgorithm 140, the color analysis algorithm 142, the pixel entropyalgorithm 144, the edge detection algorithm 146, the line detectionalgorithm 148, the boundary detection algorithm 150, the compressionanalysis algorithm 152, or the surface history algorithm 154) may beassociated with a weight factor. The weight factor may be, for example,an integer value from 0-10 or a floating point value from 0-1. Eachweight factor from each algorithm may indicate an extent to which theparticular algorithm's output confidence level is to be taken intoaccount when determining a final confidence level of matching. Forexample, the dynamically weighted confidence level algorithm 156 may beused to set, dynamically, a weight factor for each algorithm's output.The weight factors may be dynamic because certain algorithms may be moreor less effective for determining certain types of surfaces.

For example, the pixel value analysis algorithm 140 may be highlyeffective for distinguishing asphalt, brick, painted concrete, andunpainted concrete, but less effective for distinguishing mulch andgrass. Therefore, when the pixel value analysis algorithm 140 determinesthe surface type to be asphalt, brick, painted concrete, or unpaintedconcrete, a weight factor may be set in a high range, such as between0.70 and 0.95 on a 0-1 scale. By contrast, when the pixel value analysisalgorithm 140 determines the surface type to be mulch or grass, a weightfactor may be set in a low range, such as between 0.20 and 0.40 on a 0-1scale.

In another example, the line detection algorithm 148 may be veryeffective for identifying brick, but less effective for identifyingdirt. Therefore, when the line detection algorithm 148 determines thesurface type to be brick, a weight factor may be set in a high range,such as between 0.90 and 0.95 on a 0-1 scale. By contrast, when the linedetection algorithm 148 determines the surface type to be dirt, a weightfactor may be set in a low range, such as between 0.20 and 0.40 on a 0-1scale. More details of determining a final confidence level of matchingare described with reference to FIG. 9.

Referring again to FIGS. 5-6 and 7A-F, the image analysis software 114is not limited to the motion detection algorithm 138, the pixel valueanalysis algorithm 140, the color analysis algorithm 142, the pixelentropy algorithm 144, the edge detection algorithm 146, the linedetection algorithm 148, the boundary detection algorithm 150, thecompression analysis algorithm 152, the surface history algorithm 154,and the dynamically weighted probability algorithm 156, or anycombinations thereof for determining a type of surface being marked ortraversed. These algorithms are described for purposes of illustrationonly. Any algorithms that are useful for determining a type of surfacebeing marked or traversed may be implemented in the image analysissoftware 114. More details of an example of a method of using the motiondetection algorithm 138, the pixel value analysis algorithm 140, thecolor analysis algorithm 142, the pixel entropy algorithm 144, the edgedetection algorithm 146, the line detection algorithm 148, the boundarydetection algorithm 150, the compression analysis algorithm 152, thesurface history algorithm 154, and the dynamically weighted confidencelevel algorithm 156 of the image analysis software 114 for determining atype of surfacing being marked or traversed are described with referenceto FIG. 9.

Referring to FIG. 8, a functional block diagram of examples of the inputdevices 116 of the imaging-enabled marking device 100 is presented. Forexample, the input devices 116 of the imaging-enabled marking device 100of the present disclosure may include one or more of the following typesof sensors: a sonar sensor 1010, an IMU 1012, an infrared (IR) sensor1014, a temperature sensor 1016, a light sensor 1018, and a digitalaudio recorder 1020. However, it should be appreciated that other typesof sensors may also be suitable, as aspects of the present disclosurerelating to sensing are not limited to any particular types orcombinations of sensors.

Unlike the digital video cameras 112, the illustrative input devices 116are not imaging devices capable of detecting visible features of thesurface being marked or traversed. However, information from the inputdevices 116 may be used to supplement and/or support the processes ofthe image analysis software 114, such as the processes described withreference to the method 1100 of FIG. 9. For example, the input devices116 may be used to further increase a confidence level (e.g., aconfidence score indicative of a likelihood of a correct hypothesis) ofa type of surface determined by the algorithms of the image analysissoftware 114.

Referring again to FIG. 6, reference information associated with theinput devices 116 may be stored in the reference data 158. For example,the reference data 158 may include reference sonar data 176, which isassociated with the sonar sensor 1010; reference IR data 178, which isassociated with the IR sensor 1014; and reference audio data 180, whichis associated with the digital audio recorder 1020.

The sonar sensor 1010 is a device that emits an acoustic signal anddetects the acoustic signal that is reflected back from one or moreobjects. In one example, the sonar sensor 1010 may be an ultrasonicsensor that generates high frequency sound waves and evaluates an echothat is received back by the sensor. When attached to theimaging-enabled marking device 100, the sonar sensor 1010 may emit anacoustic signal toward a surface being marked or traversed and detectsthe acoustic signal that is reflected back from the surface being markedor traversed. Applicants have recognized and appreciated that differenttypes of surfaces may yield different signal strength returns andreflection characteristics because, for example, different types ofsurfaces may have different acoustic absorption characteristics. Thatis, different types of surfaces may have different sonar signatures. Aset of sonar signatures for the different types of surfaces may bestored in the reference sonar data 176 of the reference data 158. Inthis way, the sonar sensor 1010 and the reference sonar data 176 may beused to supplement and/or support any algorithms of the image analysissoftware 114.

An IMU is an electronic device that measures and reports an object'sacceleration, orientation, and/or gravitational forces by use of one ormore inertial sensors, such as one or more accelerometers, gyroscopes,and/or compasses. The IMU 1012 may be any commercially available IMUdevice for reporting the acceleration, orientation, and/or gravitationalforces of any device in which it is installed. In one example, the IMU1012 may be an IMU 6 Degrees of Freedom (6DOF) device, which isavailable from SparkFun Electronics (Boulder, Colo.). This SparkFun IMU6DOF device has Bluetooth® capability and provides 3 axes ofacceleration data, 3 axes of gyroscopic data, and 3 axes of magneticdata. In one example, information from the IMU 1012 may be used to applycertain correction to an output of the image analysis software 114 tocompensate for discrepancies due to the imaging-enabled marking device100 being held at a certain slope or angle and/or moving in a certainway. In another example, the IMU 1012 may be used to detect any motionof the imaging-enabled marking device 100, and readings from the IMU1012 may be used by the motion detection algorithm 138 to activate theone or more digital video cameras 112.

The IR sensor 1014 is an electronic device that measures infrared lightradiating from objects in its field of view. The IR sensor 1014 may beused, for example, to measure a temperature of a surface being marked ortraversed. The temperature sensor 1016 and light sensor 1018 areexamples of environmental sensors that may be used in conjunction withthe IR sensor 1014. In one example, the temperature sensor 1016 maydetect ambient temperature ranging from about −40° C. to about +125° C.,and the light sensor 1018 may be a cadmium sulfide (CdS) photocell,which is a photoresistor device whose resistance decreases when exposedto increasing incident light intensity. In this example, the data thatis returned from the light sensor 1018 is a resistance measurement.

Applicants have recognized and appreciated that, because differentsurface types may have different energy absorption characteristicsand/or specific heat capacities, certain types of surfaces may havehigher or lower temperatures compared to other types of surfaces giventhe same or similar ambient temperature levels and ambient light levels.Accordingly, the IR sensor 1014 may be used in combination with thetemperature sensor 1016 and the light sensor 1018 to determine a type ofsurface being marked or traversed. For instance, different surface typesmay have different expected temperatures depending on environmentalconditions such as ambient temperature and ambient light (e.g., sunlightlevel). Thus, each surface type may have a characteristic expectedtemperature that is close to the ambient temperature but adjusted forthe ambient light level. As a more specific example, grass may have anexpected temperature that is about equal to the ambient temperature whenshaded, but about equal to the ambient temperature plus 10° F. when inbright sunlight. By contrast, asphalt or concrete may have an expectedtemperature that is about equal to the ambient temperature when shaded,but about equal to the ambient temperature plus 30° F. when in brightsunlight. Accordingly, in some embodiments, if readings from thetemperature sensor 1016 indicate an ambient temperature of 80° F.,readings from the light sensor 1018 indicate bright sunlight, andreadings from the IR sensor 1014 indicate a surface temperature of 110°F., then there is a high probability of the surface type being asphalt,not grass.

The contents of the reference IR data 178 of the reference data 158 mayinclude a lookup table that correlates surface types to relative surfacetemperatures with respect to ambient temperatures and ambient lightlevels. In this way, the IR sensor 1014, the temperature sensor 1016,the light sensor 1018 and the reference IR data 178 may be used tosupplement and/or support any algorithms of the image analysis software114.

Additionally, the light sensor 1018 may be used to activate the lightsource 120 to illuminate a surface being marked or traversed whenambient light levels drop below a certain programmed threshold. Poorresults from the algorithms of the image analysis software 114 may alsobe used to activate the light source 120 to illuminate the surface beingmarked or traversed, as poor results may be an indication of poorlighting conditions.

Although the IR sensor 1014 is used in the above example to measure atemperature of the surface being marked or traversed, it should beappreciated that the IR sensor 1014 may also be used in other manners.For example, the data collected by the IR sensor 1014 may be used togenerate a spectral signature of the surface being marked or traversed.

More generally, one or more radiation sensors (of which the IR sensor1014 is an example) may be employed to measure an amount ofelectromagnetic radiation reflected by the sensed surface at each of oneor more selected wavelengths or ranges of wavelengths (e.g., visiblelight, infrared, ultraviolet, etc.). The source of the radiation may benatural sun light and/or an artificial light source configured tooperate in conjunction with the light sensors (e.g., a calibrated lightsource emitting light at a specific wavelength or range of wavelengths,such as a broad spectrum IR light emitting diode). Some suitable set ofinformation may be derived from the collected sensor data (e.g., apercentage of radiation reflected by the surface at each selectedwavelength or range of wavelengths) and may be used as a spectralsignature of the sensed surface. This spectral signature may be comparedagainst a list of reference spectral signatures correspondingrespectively to various surface types, to identify a potentiallymatching surface type. Further details regarding spectral signatures ofdifferent types of surfaces are discussed in “Remote Sensing Tutorial”¹by Nicholas Short, which is hereby incorporated herein by reference. ¹Available at http://rst.gsfc.nasa.gov/

FIG. 8A reproduces, from the Remote Sensing Tutorial, illustrativespectral signatures of three different surface types (namely, dry baresoil, vegetation, and water). As seen in this example, vegetation showsa small peak in the green region of the visible spectrum (e.g.,wavelengths around 0.5 μm) and strong reflectance at wavelengths between0.7 μm and 1.2 μm. On the other hand, soil shows a steadily increasingreflectance at wavelengths between 0.4 μm and 1.2 μm, from roughly 15%to 40%. These and other differences in spectral signatures may be usedto distinguish the surface types “grass” and “soil,” in accordance withsome embodiments of the present disclosure. For example, a surfacespectral signature derived based on sensor data collected from a surfacebeing marked or traversed may be compared with reference spectralsignatures of “grass,” “soil,” and/or other surface types. to identify asurface type whose spectral signature best matches the surface spectralsignature.

Applicants have further recognized and appreciated that the sound ofwalking on grass may be different from the sound of walking on pavement.Accordingly, digital audio recorder 1020 may be useful for determining atype of surface being marked or traversed. For example, different typesof surfaces may have different audio signatures. A set of referenceaudio signatures for the different types of surfaces may be stored inthe reference audio data 180 of reference data 158. In this way, thedigital audio recorder 1020 may be used to supplement and/or support anyalgorithms of the image analysis software 114. The digital audiorecorder 1020 may be, for example, any standard digital audio recordingdevice. The digital audio output may be stored in any standard and/orproprietary audio file format (e.g., WAV, MP3, etc.).

Referring again to FIG. 8, certain other algorithms (not shown) may beincorporated in, for example, the image analysis software 114 forprocessing the readings from any types of the input devices 116 that areinstalled in the imaging-enabled marking device 100 and determining aconfidence level of matching for each type of the input devices 116 orvarious combinations of the input devices 116.

Referring to FIG. 11, a flow diagram of an example of a method 1300 fordetermining a type of surfacing being marked or traversed by theimaging-enabled marking device 100 is presented. The method 110 mayinclude, but is not limited to, the following steps.

At step 1301, frames of video are captured. These captured frames ofvideo data are then analyzed at step 1302 in one or more analysisprocesses. Each of the analysis processes provides a putativeidentification result and an associated confidence level. Severalexemplary analysis processes are discussed below with reference to FIG.9. At step 1303, the results of the analyses are used to generate adynamic weight factor for a confidence level of each of the analyses.The confidence levels and weight factors for each of the analyses arethen processed at step 1304 to generate a final confidence levelcorresponding to an identification result. Further details relating tothese steps are discussed in further detail below with respect to anexemplary embodiment with reference to FIG. 9.

Referring to FIG. 9, a flow diagram of an example of a method 1100 ofusing a camera system and image analysis software for determining a typeof surfacing being marked or traversed by the imaging-enabled markingdevice 100 is presented. The method 1100 may include, but is not limitedto, the following steps, which are not limited to being performed in anyparticular order.

At step 1110, the starting of the motion of the imaging-enabled markingdevice 100 may be sensed by the motion detection algorithm 138 and thedigital video camera 112 may be activated. More specifically, the motiondetection algorithm 138 may monitor, for example, readings from the IMU1012 and/or the output of an optical flow algorithm to determine thebeginning of any motion of the imaging-enabled marking device 100. Whenthe starting motion is sensed, the digital video camera 112 may beactivated.

At step 1112, the ending of the motion of the imaging-enabled markingdevice 100 may be sensed by the motion detection algorithm 138 and thedigital video camera 112 may be deactivated. More specifically, themotion detection algorithm 118 may monitor, for example, readings fromthe IMU 1012 and/or the output of an optical flow algorithm to determinethe ending of any motion of the imaging-enabled marking device 100. Whenthe ending motion is sensed, the digital video camera 112 may bedeactivated.

At step 1114, certain frames of the digital video clip that was takenwhile the imaging-enabled marking device 100 was in motion are stored.For example, every n^(th) frame (e.g., every 5^(th), 10^(th) or 20^(th)frame) of the image data 134 from the digital video camera 112 may bepassed to the processing unit 122 and stored in the local memory 124.Each frame of the image data 134 may be time-stamped and/or geo-stamped.For example, each frame of the image data 134 may be encoded withcurrent date and time from the processing unit 122 and/or currentgeo-location data from the location tracking system 130.

At step 1116, a pixel value analysis may be performed on one or moreframes of the image data 134 and a confidence level of matching may bedetermined based on grayscale luminance distribution. For example, thepixel value analysis algorithm 140 may process the current frames of theimage data 134 and generate a grayscale luminance distribution histogramof the current surface being marked or traversed. The pixel valueanalysis algorithm 140 may then compare the current grayscale luminancedistribution histogram to the reference histogram data 160 stored in thereference data 158. Examples of reference histograms to which thecurrent grayscale luminance distribution histogram may be compared areshown in FIGS. 7A-F and described above. The output of the pixel valueanalysis algorithm 140 may include a confidence level for each surfacetype indicative of an extent to which the current surface matches thatsurface type. In one example, the confidence levels based on grayscaleluminance distribution are: ASPHALT=62%, MULCH=7%, BRICK=81%, GRASS=5%,PAINTED CONCRETE=54%, UNPAINTED CONCRETE=35%, and DIRT=42%.

In this example, the type of surface having the highest confidence levelof matching with respect to grayscale luminance distributions is BRICK.This information may be time-stamped, geo-stamped, and stored in thesurface type data 136. The results of this step may be used to confirm,validate, verify, and/or otherwise support the analyses performed in anyother steps of the method 1100.

At step 1118, a pixel entropy analysis may be performed on one or moreframes of image data 134 and a confidence level of matching may bedetermined based on a degree of randomness. For example, the pixelentropy algorithm 144 may process the current frames of the image data134 and generate an average entropy value of the current surface beingmarked or traversed. In one example, the pixel entropy algorithm 144determines the average entropy value of the current frames of the imagedata 134 to be about 6.574. The pixel entropy algorithm 144 thencompares the current average entropy value of 6.574 to the referenceentropy values 164 stored in the reference data 158 (e.g., as shown inTable 2). The output of the pixel entropy algorithm 144 may include aconfidence level for each surface type indicative of an extent to whichthe current surface matches that surface type. In one example, theconfidence levels based on degree of randomness are: ASPHALT=73%,MULCH=24%, BRICK=89%, GRASS=31%, PAINTED CONCRETE=9%, UNPAINTEDCONCRETE=49%, and DIRT=26%.

In this example, the type of surface having the highest confidence levelof matching with respect to degree of randomness is BRICK. Thisinformation may be time-stamped, geo-stamped, and stored in the surfacetype data 136. The results of this step may be used to confirm,validate, verify, and/or otherwise support the analyses performed in anyother steps of the method 1100.

At step 1120, a color analysis may be performed on one or more frames ofthe image data 134 and a confidence level of matching is determinedbased on a most prevalent color. For example, the color analysisalgorithm 142 may process the current frames of image data 134 andgenerate a most prevalent color of the current surface being marked ortraversed. In one example, the color analysis algorithm 142 determinesthe most prevalent color present in the current frames of the image data134 is LIGHT BROWN. The color analysis algorithm 142 then compares thecurrent most prevalent color of LIGHT BROWN to the reference color data162 stored in the reference data 158 (e.g., as shown in Table 1). Theoutput of color analysis algorithm 142 may include a confidence levelfor each surface type indicative of an extent to which the currentsurface matches that surface type. In one example, the confidence levelsbased on the most prevalent color detected in the current frames ofimages are: ASPHALT=82%, MULCH=43%, BRICK=57%, GRASS=11%, PAINTEDCONCRETE=9%, UNPAINTED CONCRETE=75%, and DIRT=91%.

In this example, the type of surface having the highest confidence levelof matching with respect to most prevalent color is DIRT. Thisinformation may be time-stamped, geo-stamped, and stored in the surfacetype data 136. The results of this step may be used to confirm,validate, verify, and/or otherwise support the analyses performed in anyother steps of the method 1100.

At step 1122, further color analysis may be performed on one or moreframes of image data 134 and a confidence level of matching may bedetermined based on hue and saturation. For example, the color analysisalgorithm 142 may process the current frames of image data 134 andgenerate the hue and saturation of the current surface being marked ortraversed. The color analysis algorithm 142 then compares the currenthue and saturation to the reference hue and saturation data 166 storedin the reference data 158. The output of the color analysis algorithm142 may include a confidence level for each surface type indicative ofan extent to which the current surface matches that surface type. In oneexample, the confidence levels based on hue and saturation are:ASPHALT=27%, MULCH=59%, BRICK=74%, GRASS=11%, PAINTED CONCRETE=9%,UNPAINTED CONCRETE=33%, and DIRT=46%.

In this example, the type of surface having the highest confidence levelwith respect to hue and saturation is BRICK. This information may betime-stamped, geo-stamped, and stored in the surface type data 136. Theresults of this step may be used to confirm, validate, verify, and/orotherwise support the analyses performed in any other steps of themethod 1100.

At step 1124, an edge detection analysis may be performed on one or moreframes of the image data 134 and a confidence level of matching may bedetermined based on edge characteristics. For example, the edgedetection algorithm 146 may process the current frames of image data 134and generate edge characteristics of the current surface being marked ortraversed. The edge detection algorithm 146 then compares the currentedge characteristics to the reference edge data 168 stored in thereference data 158. The output of edge detection algorithm 146 mayinclude a confidence level of matching value for the current surfacematching each surface type. In one example, the confidence levels basedon edge characteristics are: ASPHALT=73%, MULCH=12%, BRICK=35%,GRASS=11%, PAINTED CONCRETE=9%, UNPAINTED CONCRETE=67%, and DIRT=91%.

In this example, the type of surface having the highest confidence levelwith respect to edge characteristics is DIRT. This information may betime-stamped, geo-stamped, and stored in the surface type data 136. Theresults of this step may be used to confirm, validate, verify, and/orotherwise support the analyses performed in any other steps of themethod 1100.

At step 1126, a line detection analysis may be performed on one or moreframes of image data 134 and a confidence level of matching may bedetermined based on line characteristics. For example, the linedetection algorithm 148 may process the current frames of image data 134and generate line characteristics of the current surface being marked ortraversed. The line detection algorithm 148 then compares the currentline characteristics to the reference line data 170 stored in thereference data 158. The output of line detection algorithm 148 mayinclude a confidence level for each surface type indicative of an extentto which the current surface matches that surface type. In one example,the confidence levels based on line characteristics are: ASPHALT=25%,MULCH=19%, BRICK=94%, GRASS=16%, PAINTED CONCRETE=9%, UNPAINTEDCONCRETE=31%, and DIRT=17%.

In this example, the type of surface having the highest confidence levelwith respect to line characteristics is BRICK. This information may betime-stamped, geo-stamped, and stored in the surface type data 136. Theresults of this step may be used to confirm, validate, verify, and/orotherwise support the analyses performed in any other steps of themethod 1100.

At step 1128, a compression analysis may be performed on one or moreframes of image data 134 and a confidence level of matching may bedetermined based on the achieved compression ratio. For example, thecompression analysis algorithm 152 performs standard JPEG compressionoperations on the current frames of image data 134 and generates acompression ratio for the current surface being marked or traversed. Thecompression analysis algorithm 152 then compares the current compressionratio to the reference compression data 174 stored in the reference data158 (e.g., as shown in Table 3). The output of the compression analysisalgorithm 152 may include a confidence level for each surface typeindicative of an extent to which the current surface matches thatsurface type. In one example, the confidence levels based on achievedcompression ratio are: ASPHALT=25%, MULCH=19%, BRICK=27%, GRASS=16%,PAINTED CONCRETE=23%, UNPAINTED CONCRETE=31%, and DIRT=13%.

In this example, the type of surface having the highest confidence levelwith respect to compression ratio is CONCRETE. This information may betime-stamped, geo-stamped, and stored in the surface type data 136. Theresults of this step may be used to confirm, validate, verify, and/orotherwise support the analyses performed in any other steps of themethod 1100.

At step 1130, a candidate current surface type (e.g., as determined inany of the steps described above) may be compared to candidate surfacetypes of previous frames of image data 134. For example, the surfacehistory algorithm 154 queries the surface type data 136 for confidencelevels from any of the aforementioned analyses performed in the method1100. The surface history algorithm 154 determines what surface type ismost often indicated as having a highest confidence level of matching inone or more previous frames of the image data 134. The surface historyalgorithm 154 may then indicate that the current frame of the image data134 is most likely to be this same surface type. In one example, thesurface history algorithm 154 determines that BRICK is the surface typemost often indicated as having a highest confidence level of matching inone or more previous frames of the image data 134. Therefore, thesurface history algorithm 154 indicates that the surface type of thecurrent frame of the image data 134 is most likely to be BRICK. Thisinformation is time-stamped, geo-stamped, and stored in the surface typedata 136. The results of this step may be used to confirm, validate,verify, and/or otherwise support the analyses performed in any othersteps of the method 1100.

At step 1132, a boundary detection analysis may be performed on one ormore frames of the image data 134 and a confidence level of matching maybe determined for two or more subsections of a current frame of imagedata 134. In this step, the boundary detection algorithm 150 may beexecuted only when a low confidence level is returned for all surfacetypes in the analyses of substantially all other steps of the method1100. In one example, the boundary detection algorithm 150 is executedonly when a confidence level of less than 50% is returned for allsurface types in the analyses of substantially all other steps of themethod 1100.

When the boundary detection analysis is performed, the boundarydetection algorithm 150 may be used to analyze two or more subsectionsof a frame of the image data 134 to determine whether the frame is a“multi-surface” frame. For example, the boundary detection algorithm 150may analyze each of the two or more subsections using any image analysisprocesses of the present disclosure for determining a type of surfacefound in any of the two or more subsections. The output of the boundarydetection algorithm 150 may include a confidence level for eachsubsection of the current frame of the image data 134. When two or moreframe subsections indicate a high probability of different surfacetypes, the boundary detection algorithm may identify the frame as a“multi-surface” frame (e.g., a framing containing a boundary between twoor more surface types).

At step 1134, the image analysis software 114 may process readings fromany types of input devices 116 to confirm, validate, verify, and/orotherwise support the analyses performed in any other steps of themethod 1100. For example, the image analysis software 114 may determinea confidence level for each type of input device 116 or any combinationsof input devices 116. In one example, the image analysis software 114may process readings from the sonar sensor 1010 and compare the currentreadings to information in the reference sonar data 176. In anotherexample, the image analysis software 114 may process readings from theIR sensor 1014, the temperature sensor 1016, and the light sensor 1018and compare the current combination of these readings to information inthe reference IR data 178. In yet another example, the image analysissoftware 114 may compare the digital audio captured by the digital audiorecorder 1020 to information in the reference audio data 180.

At step 1136, a dynamic weight factor may be generated for eachconfidence level associated with each analysis performed in the method1100. For example, the dynamically weighted confidence level algorithm156 generates respective dynamic weight factors for any outputs of thepixel value analysis algorithm 140, the color analysis algorithm 142,the pixel entropy algorithm 144, the edge detection algorithm 146, theline detection algorithm 148, the boundary detection algorithm 150, thecompression analysis algorithm 152, and the surface history algorithm154. Additionally, the dynamically weighted confidence level algorithm156 may generate respective dynamic weight factors for any processingwith respect to any one or more input devices 116. Examples of dynamicweight factors for the example outputs of steps 1116, 1118, 1120, 1122,1124, 1126, and 1128 are shown in Table 4 below.

At step 1138, all confidence levels and weight factors may be processedand a final confidence level may be generated. For example, thedynamically weighted confidence level algorithm 156 processes anyoutputs of the pixel value analysis algorithm 140, the color analysisalgorithm 142, the pixel entropy algorithm 144, the edge detectionalgorithm 146, the line detection algorithm 148, the boundary detectionalgorithm 150, the compression analysis algorithm 152, and the surfacehistory algorithm 154, along with the weight factors generated in thestep 1136 and generates a final confidence level. Table 4 below shows anexample of the outputs and dynamic weight factors of steps 1116, 1118,1120, 1122, 1124, 1126, and 1128 and a final confidence level. Table 4indicates only the surface type with the highest confidence level foreach of the aforementioned steps.

TABLE 4 Example algorithm outcomes and dynamic weight factors DynamicWeight Surface Confidence Factor Surface Type Type Level (0-1 scale)Pixel value analysis of step 1116 Brick 81% 0.60 Entropy analysis ofstep 1118 Brick 89% 0.91 Most prevalent color analysis of Dirt 91% 0.80step 1120 Hue and saturation analysis of Brick 74% 0.50 step 1122 Edgedetection analysis of Dirt 91% 0.75 step 1124 Line detection analysis ofBrick 94% 0.99 step 1126 Compression analysis of step 1128 Concrete 31%0.20 Surface Type Result = Brick 2.59/4.75 n/a

In one example, referring to Table 4, the “surface type result” may becalculated by multiplying the confidence level by the dynamic weightfactor for each entry. Next, the aforementioned calculations for allentries for each respective surface type are summed together. “Surfacetype result” may be the surface type with the highest sum. For example,referring to Table 4, there are four entries for BRICK with thefollowing results:(81%×0.60)+(89%×0.91)+(74%×0.50)+(94%×0.99)=48%+81%+37%+93%=2.59 (out of4.75). There are two entries for DIRT with the following results:(91%×0.80)+(91%×0.75)=73%+68%=1.41 (out of 4.75). There is one entry forCONCRETE with the following results: (31%×0.20)=0.06 (out of 4.75). Inthis example, BRICK has the highest total and, thus, the “surface typeresult” is BRICK at 2.59 (out of 4.75).

The “surface type result” may be calculated for each frame of the imagedata 134 that is processed. Once the final “surface type result” isdetermined, it may be time-stamped and geo-stamped and stored in thesurface type data 136 for each frame of the image data 134. The contentsof the surface type data 136 may be included in any electronic recordsof locate operations.

The method 1100 provides an example of running multiple image analysisprocesses, wherein running each analysis may serve to confirm, validate,verify, and/or otherwise support the results of any other analyses and,thereby, increase the probability of correctly determining the type ofsurface. That is, while executing any one image analysis process alonemay provide a certain amount of confidence (e.g., as expressed in termsof confidence levels or scores) in the surface type that is determined,running multiple image analysis processes may serve to increase theconfidence (e.g., increase the confidence level or score of matching) ofsurface type that is determined.

Additionally, the method 1100 is not limited to executing theaforementioned number and types of image analyses for determining a typeof surface being marked or traversed. Any number and types of imageanalysis processes may be executed in any suitable order in the method1100. Further, the image analysis processes of the method 1100 may beexecuted in real time during locate operations for determining andrecording the types of surfaces marked and/or traversed. Alternatively,post processing of frames of the image data 134 from one or more of thedigital video cameras 112 may occur for determining and recording thetypes of surfaces.

Referring to FIG. 10, a functional block diagram of an example of alocate operations system 1200 that includes a network of imaging-enabledmarking devices 100 is presented. The locate operations system 1200 mayinclude any number of imaging-enabled marking devices 100 that areoperated by, for example, respective locate personnel 1210. Examples oflocate personnel 1210 include locate technicians. Associated with eachlocate personnel 1210 and/or imaging-enabled marking device 100 may bean onsite computer 1212. Therefore, the locate operations system 1200may also include any number of onsite computers 1212.

Each onsite computer 1212 may be any suitable computing device, such as,but not limited to, a computer that is present in a vehicle that isbeing used by locate personnel 1210 in the field. For example, an onsitecomputer 1212 may be a portable computer, a personal computer, a laptopcomputer, a tablet device, a personal digital assistant (PDA), acellular radiotelephone, a mobile computing device, a touch-screendevice, a touchpad device, or generally any device including, orconnected to, a processor. Each imaging-enabled marking device 100 maycommunicate via a communication interface 126 with its respective onsitecomputer 1212. For instance, each imaging-enabled marking device 100 maytransmit image data 134 to its respective onsite computer 1212.

While an instance of the image analysis software 114 that includes thealgorithms described in FIG. 5, the surface type data 136, and thereference data 158 may reside and operate at each imaging-enabledmarking device 100, an instance of the image analysis software 114, thesurface type data 136, and the reference data 158 may also reside ateach onsite computer 1212. In this way, the image data 134 may beprocessed at the onsite computer 1212 in addition to, or instead of,being processed at the imaging-enabled marking device 100. Additionally,the onsite computer 1212 may process the image data 134 concurrentlywith the imaging-enabled marking device 100.

Additionally, the locate operations system 1200 may include a centralserver 1214. The central server 1214 may be a centralized computer, suchas a central server of, for example, an underground facility locateservice provider. One or more networks 1216 provide a communicationmedium by which information may be exchanged between the imaging-enabledmarking devices 100, the onsite computers 1212, and/or the centralserver 1214. The networks 1216 may include, for example, any local areanetwork (LAN), wide area network (WAN), and/or the Internet. Theimaging-enabled marking devices 100, the onsite computers 1212, and/orthe central server 1214 may be connected to the networks 1216 using anywired and/or wireless networking technologies.

While an instance of the image analysis software 114, the surface typedata 136, and the reference data 158 may reside and operate at eachimaging-enabled marking device 100 and/or at each onsite computer 1212,an instance of the image analysis software 114, the surface type data136, and the reference data 158 may also reside at the central server1214. In this way, the image data 134 may be process at the centralserver 1214 in addition to, or instead of, at each imaging-enabledmarking device 100 and/or at each onsite computer 1212. Additionally,the central server 1214 may process the image data 134 concurrently withthe imaging-enabled marking devices 100 and/or the onsite computers1212.

While various inventive embodiments have been described and illustratedherein, those of ordinary skill in the art will readily envision avariety of other means and/or structures for performing the functionand/or obtaining the results and/or one or more of the advantagesdescribed herein, and each of such variations and/or modifications isdeemed to be within the scope of the inventive embodiments describedherein. More generally, those skilled in the art will readily appreciatethat all parameters, dimensions, materials, and configurations describedherein are meant to be exemplary and that the actual parameters,dimensions, materials, and/or configurations will depend upon thespecific application or applications for which the inventive teachingsis/are used. Those skilled in the art will recognize, or be able toascertain using no more than routine experimentation, many equivalentsto the specific inventive embodiments described herein. It is,therefore, to be understood that the foregoing embodiments are presentedby way of example only and that, within the scope of the appended claimsand equivalents thereto, inventive embodiments may be practicedotherwise than as specifically described and claimed. Inventiveembodiments of the present disclosure are directed to each individualfeature, system, article, material, kit, and/or method described herein.In addition, any combination of two or more such features, systems,articles, materials, kits, and/or methods, if such features, systems,articles, materials, kits, and/or methods are not mutually inconsistent,is included within the inventive scope of the present disclosure.

The above-described embodiments can be implemented in any of numerousways. For example, the embodiments may be implemented using hardware,software or a combination thereof. When implemented in software, thesoftware code can be executed on any suitable processor or collection ofprocessors, whether provided in a single computer or distributed amongmultiple computers.

Further, it should be appreciated that a computer may be embodied in anyof a number of forms, such as a rack-mounted computer, a desktopcomputer, a laptop computer, or a tablet computer. Additionally, acomputer may be embedded in a device not generally regarded as acomputer but with suitable processing capabilities, including a PersonalDigital Assistant (PDA), a smart phone or any other suitable portable orfixed electronic device.

Also, a computer may have one or more input and output devices. Thesedevices can be used, among other things, to present a user interface.Examples of output devices that can be used to provide a user interfaceinclude printers or display screens for visual presentation of outputand speakers or other sound generating devices for audible presentationof output. Examples of input devices that can be used for a userinterface include keyboards, and pointing devices, such as mice, touchpads, and digitizing tablets. As another example, a computer may receiveinput information through speech recognition or in other audible format.

Such computers may be interconnected by one or more networks in anysuitable form, including a local area network or a wide area network,such as an enterprise network, and intelligent network (IN) or theInternet. Such networks may be based on any suitable technology and mayoperate according to any suitable protocol and may include wirelessnetworks, wired networks or fiber optic networks.

As a more specific example, an illustrative computer that may be usedfor surface type detection in accordance with some embodiments comprisesa memory, one or more processing units (also referred to herein simplyas “processors”), one or more communication interfaces, one or moredisplay units, and one or more user input devices. The memory maycomprise any computer-readable media, and may store computerinstructions (also referred to herein as “processor-executableinstructions”) for implementing the various functionalities describedherein. The processing unit(s) may be used to execute the instructions.The communication interface(s) may be coupled to a wired or wirelessnetwork, bus, or other communication means and may therefore allow theillustrative computer to transmit communications to and/or receivecommunications from other devices. The display unit(s) may be provided,for example, to allow a user to view various information in connectionwith execution of the instructions. The user input device(s) may beprovided, for example, to allow the user to make manual adjustments,make selections, enter data or various other information, and/orinteract in any of a variety of manners with the processor duringexecution of the instructions.

The various methods or processes outlined herein may be coded assoftware that is executable on one or more processors that employ anyone of a variety of operating systems or platforms. Additionally, suchsoftware may be written using any of a number of suitable programminglanguages and/or programming or scripting tools, and also may becompiled as executable machine language code or intermediate code thatis executed on a framework or virtual machine.

In this respect, various inventive concepts may be embodied as acomputer readable storage medium (or multiple computer readable storagemedia) (e.g., a computer memory, one or more floppy discs, compactdiscs, optical discs, magnetic tapes, flash memories, circuitconfigurations in Field Programmable Gate Arrays or other semiconductordevices, or other non-transitory medium or tangible computer storagemedium) encoded with one or more programs that, when executed on one ormore computers or other processors, perform methods that implement thevarious embodiments of the invention discussed above. The computerreadable medium or media can be transportable, such that the program orprograms stored thereon can be loaded onto one or more differentcomputers or other processors to implement various aspects of thepresent invention as discussed above.

The terms “program” or “software” are used herein in a generic sense torefer to any type of computer code or set of computer-executableinstructions that can be employed to program a computer or otherprocessor to implement various aspects of embodiments as discussedabove. Additionally, it should be appreciated that according to oneaspect, one or more computer programs that when executed perform methodsof the present invention need not reside on a single computer orprocessor, but may be distributed in a modular fashion amongst a numberof different computers or processors to implement various aspects of thepresent invention.

Computer-executable instructions may be in many forms, such as programmodules, executed by one or more computers or other devices. Generally,program modules include routines, programs, objects, components, datastructures, etc. that perform particular tasks or implement particularabstract data types. Typically the functionality of the program modulesmay be combined or distributed as desired in various embodiments.

Also, data structures may be stored in computer-readable media in anysuitable form. For simplicity of illustration, data structures may beshown to have fields that are related through location in the datastructure. Such relationships may likewise be achieved by assigningstorage for the fields with locations in a computer-readable medium thatconvey relationship between the fields. However, any suitable mechanismmay be used to establish a relationship between information in fields ofa data structure, including through the use of pointers, tags or othermechanisms that establish relationship between data elements.

Also, various inventive concepts may be embodied as one or more methods,of which an example has been provided. The acts performed as part of themethod may be ordered in any suitable way. Accordingly, embodiments maybe constructed in which acts are performed in an order different thanillustrated, which may include performing some acts simultaneously, eventhough shown as sequential acts in illustrative embodiments.

All definitions, as defined and used herein, should be understood tocontrol over dictionary definitions, definitions in documentsincorporated by reference, and/or ordinary meanings of the definedterms.

The indefinite articles “a” and “an,” as used herein in thespecification and in the claims, unless clearly indicated to thecontrary, should be understood to mean “at least one.”

The phrase “and/or,” as used herein in the specification and in theclaims, should be understood to mean “either or both” of the elements soconjoined, i.e., elements that are conjunctively present in some casesand disjunctively present in other cases. Multiple elements listed with“and/or” should be construed in the same fashion, i.e., “one or more” ofthe elements so conjoined. Other elements may optionally be presentother than the elements specifically identified by the “and/or” clause,whether related or unrelated to those elements specifically identified.Thus, as a non-limiting example, a reference to “A and/or B”, when usedin conjunction with open-ended language such as “comprising” can refer,in one embodiment, to A only (optionally including elements other thanB); in another embodiment, to B only (optionally including elementsother than A); in yet another embodiment, to both A and B (optionallyincluding other elements); etc.

As used herein in the specification and in the claims, “or” should beunderstood to have the same meaning as “and/or” as defined above. Forexample, when separating items in a list, “or” or “and/or” shall beinterpreted as being inclusive, i.e., the inclusion of at least one, butalso including more than one, of a number or list of elements, and,optionally, additional unlisted items. Only terms clearly indicated tothe contrary, such as “only one of” or “exactly one of,” or, when usedin the claims, “consisting of,” will refer to the inclusion of exactlyone element of a number or list of elements. In general, the term “or”as used herein shall only be interpreted as indicating exclusivealternatives (i.e. “one or the other but not both”) when preceded byterms of exclusivity, such as “either,” “one of,” “only one of,” or“exactly one of” “Consisting essentially of,” when used in the claims,shall have its ordinary meaning as used in the field of patent law.

As used herein in the specification and in the claims, the phrase “atleast one,” in reference to a list of one or more elements, should beunderstood to mean at least one element selected from any one or more ofthe elements in the list of elements, but not necessarily including atleast one of each and every element specifically listed within the listof elements and not excluding any combinations of elements in the listof elements. This definition also allows that elements may optionally bepresent other than the elements specifically identified within the listof elements to which the phrase “at least one” refers, whether relatedor unrelated to those elements specifically identified. Thus, as anon-limiting example, “at least one of A and B” (or, equivalently, “atleast one of A or B,” or, equivalently “at least one of A and/or B”) canrefer, in one embodiment, to at least one, optionally including morethan one, A, with no B present (and optionally including elements otherthan B); in another embodiment, to at least one, optionally includingmore than one, B, with no A present (and optionally including elementsother than A); in yet another embodiment, to at least one, optionallyincluding more than one, A, and at least one, optionally including morethan one, B (and optionally including other elements); etc.

In the claims, as well as in the specification above, all transitionalphrases such as “comprising,” “including,” “carrying,” “having,”“containing,” “involving,” “holding,” “composed of,” and the like are tobe understood to be open-ended, i.e., to mean including but not limitedto. Only the transitional phrases “consisting of” and “consistingessentially of” shall be closed or semi-closed transitional phrases,respectively, as set forth in the United States Patent Office Manual ofPatent Examining Procedures, Section 2111.03.

1. An apparatus for determining a surface type of a surface on whichmarking material is to be dispensed by a marking device to mark apresence or an absence of at least one underground facility within a digarea, wherein at least a portion of the dig area is planned to beexcavated or disturbed during excavation activities, the apparatuscomprising: at least one communication interface; at least one memory tostore processor-executable instructions; and at least one processorcommunicatively coupled to the at least one memory and the at least onecommunication interface, wherein, upon execution of theprocessor-executable instructions, the at least one processor: A)obtains sensor data relating to the surface to be marked, the sensordata being collected by one or more sensors attached to the markingdevice; B) retrieves reference data associated with a plurality ofsurface types; and C) generates surface type information based at leastin part on the sensor data and the reference data.
 2. The apparatus ofclaim 1, wherein in C), the at least one processor: C1) identifies,based at least in part on the sensor data and the reference data, atleast one of the plurality of surface types as a candidate surface typefor the surface to be marked.
 3. The apparatus of claim 2, wherein thereference data comprises a plurality of reference signatures, eachreference signature being associated with at least one correspondingsurface type of the plurality of surface types, and wherein in C1), theat least one processor: C2) determines, based at least in part on thesensor data, a surface signature associated with the surface to bemarked; and C3) identifies at least one reference signature as being aclosest match with the surface signature, wherein the at least onesurface type corresponding to the at least one reference signature isidentified as the candidate surface type in C1).
 4. The apparatus ofclaim 1, wherein in C), the at least one processor: C1) identifies,based at least in part on the sensor data and the reference data, eachof the plurality of surface types as being unlikely to match the surfaceto be marked.
 5. The apparatus of claim 1, wherein the reference datacomprises a plurality of reference signatures, each reference signaturebeing associated with at least one corresponding surface type of theplurality of surface types, and wherein in C), the at least oneprocessor: C1) determines, based at least in part on the sensor data, atleast one surface signature associated with the surface to be marked;and C2) associates at least one confidence score with each one of theplurality of surface types, based at least in part on an extent to whichthe at least one surface signature matches at least one referencesignature associated with the one of the plurality of surface types. 6.The apparatus of claim 5, wherein in C), the at least one processor: C3)identifies at least one of the plurality of surface types as beingunlikely to match the surface to be marked, based at least in part on atleast one confidence score associated with the at least one of theplurality of surface types.
 7. The apparatus of claim 5, wherein the atleast one surface signature comprises first and second surfacesignatures, and where in C), the at least one processor: C3) identifies,based at least in part on the first surface signature and the referencedata, a first surface type of the plurality of surface types as possiblymatching the surface to be marked, the first surface type beingassociated with a first confidence score; C4) identifies, based at leastin part on the second surface signature and the reference data, a secondsurface type of the plurality of surface types as possibly matching thesurface to be marked, the second surface type being associated with asecond confidence score; and C5) identifies one of the first and secondsurface types as a candidate surface type for the surface to be marked,based at least in part on the first and second confidence scores.
 8. Theapparatus of claim 7, wherein the sensor data comprises first sensordata collected by at least a first sensor and second sensor datacollected by at least a second sensor, and wherein the first surfacesignature is determined based at least in part on the first sensor dataand the second surface signature is determined based at least in part onthe second sensor data.
 9. The apparatus of claim 2, wherein the one ormore sensors comprise at least one camera and the sensor data comprisesimage data representing at least one image of the surface to be markedcaptured by the camera.
 10. The apparatus of claim 9, wherein in C1),the at least one processor: C2) determines a surface signatureassociated with the surface to be marked, based at least in part on aluminance distribution associated with the image data; and C3) comparesthe surface signature with a plurality of reference signatures in thereference data to identify the candidate surface type, wherein eachreference signature is indicative of a characteristic luminancedistribution for at least one corresponding surface type in theplurality of surface types.
 11. The apparatus of claim 9, wherein inC1), the at least one processor: C2) determines a surface signatureassociated with the surface to be marked, based at least in part oncolor information associated with the image data; and C3) compares thesurface signature with a plurality of reference signatures in thereference data to identify the candidate surface type, wherein eachreference signature is indicative of a color characteristic for at leastone corresponding surface type in the plurality of surface types. 12.The apparatus of claim 11, wherein the color information associated withthe image data comprises a spatial distribution of color in the at leastone image.
 13. The apparatus of claim 11, wherein the color informationassociated with the image data comprises a most prevalent color in theat least one image.
 14. The apparatus of claim 11, wherein the colorinformation associated with the image data comprises an average color inthe at least one image.
 15. The apparatus of claim 9, wherein in C1),the at least one processor: C2) determines a surface signatureassociated with the surface to be marked, based at least in part on anentropy value derived based on the image data; and C3) compares thesurface signature with a plurality of reference signatures in thereference data to identify the candidate surface type, wherein eachreference signature is indicative of a characteristic entropy value forat least one corresponding surface type in the plurality of surfacetypes.
 16. The apparatus of claim 15, wherein the entropy value isderived based at least in part on intensity information in the imagedata.
 17. The apparatus of claim 15, wherein the entropy value isderived based at least in part on color information in the image data.18. The apparatus of claim 9, wherein in C1), the at least oneprocessor: C2) determines a surface signature associated with thesurface to be marked, based at least in part on a frequency at whichedges are detected in the at least one image; and C3) compares thesurface signature with a plurality of reference signatures in thereference data to identify the candidate surface type, wherein eachreference signature is indicative of a characteristic frequency of edgesfor at least one corresponding surface type in the plurality of surfacetypes.
 19. The apparatus of claim 9, wherein in C1), the at least oneprocessor: C2) determines a surface signature associated with thesurface to be marked, based at least in part on a presence or absence ofone or more line patterns in the at least one image; and C3) comparesthe surface signature with a plurality of reference signatures in thereference data to identify the candidate surface type, wherein eachreference signature is indicative of one or more characteristic linepatterns for at least one corresponding surface type in the plurality ofsurface types.
 20. The apparatus of claim 9, wherein in C1), the atleast one processor: C2) determines a first surface signature associatedwith the surface to be marked, based at least in part on a first portionof the at least one image; C3) compares the first surface signature witha first plurality of reference signatures in the reference data toidentify a first surface type as possibly matching the first portion ofthe at least one image; C4) determines a second surface signatureassociated with the surface to be marked, based at least in part on asecond portion of the at least one image; C5) compares the secondsurface signature with a second plurality of reference signatures in thereference data to identify a second surface type as possibly matchingthe second portion of the at least one image; and C6) determines whetherthe first surface type is different from the second surface type. 21.The apparatus of claim 9, wherein in C1), the at least one processor:C2) determines a surface compression ratio associated with the surfaceto be marked, at least in part by performing a compression operation onat least a portion of the image data; and C3) compares the surfacecompression ratio with a plurality of reference compression ratios inthe reference data to identify the candidate surface type, wherein eachreference compression ratio is associated with at least onecorresponding surface type in the plurality of surface types.
 22. Theapparatus of claim 2, wherein the one or more sensors comprise an IRsensor and the sensor data comprises at least one temperaturemeasurement of the surface to be marked taken by the IR sensor, andwherein in C1), the at least one processor: C2) compares the at leastone temperature measurement with a plurality of reference temperaturesin the reference data to identify the candidate surface type, whereineach reference temperature is associated with at least one correspondingsurface type in the plurality of surface types.
 23. The apparatus ofclaim 22, wherein the one or more sensors further comprise an ambienttemperature sensor and an ambient light sensor, and the sensor datafurther comprises an ambient temperature level and an ambient lightlevel measured respectively by the temperature sensor and an ambientlight sensor, and wherein the plurality of reference temperatures areselected based at least in part on the ambient temperature level and theambient light level.
 24. The apparatus of claim 2, wherein the one ormore sensors comprises at least one radiation sensor capable ofdetecting electromagnetic radiation in at least one frequency band, andwherein in C1), the at least one processor: C2) determines, based atleast in part on the sensor data, a surface spectral signatureassociated with the surface to be marked; and C3) compares the surfacespectral signature with a plurality of reference spectral signatures inthe reference data to identify the candidate surface type, wherein eachreference spectral signature is associated with at least onecorresponding surface type in the plurality of surface types.
 25. In asystem comprising at least one communication interface, at least onememory to store processor-executable instructions, and at least oneprocessor communicatively coupled to the at least one memory and the atleast one communication interface, a method for determining a surfacetype of a surface on which marking material is to be dispensed by amarking device to mark a presence or an absence of at least oneunderground facility within a dig area, wherein at least a portion ofthe dig area is planned to be excavated or disturbed during excavationactivities, the method comprising acts of: A) obtaining sensor datarelating to the surface to be marked, the sensor data being collected byone or more sensors attached to the marking device; B) retrieving, fromthe at least one memory, reference data associated with a plurality ofsurface types; and C) using the at least one processor to generatesurface type information based at least in part on the sensor data andthe reference data.
 26. At least one non-transitory computer-readablestorage medium encoded with at least one program includingprocessor-executable instructions that, when executed by at least oneprocessor, perform a method for determining a surface type of a surfaceon which marking material is to be dispensed by a marking device to marka presence or an absence of at least one underground facility within adig area, wherein at least a portion of the dig area is planned to beexcavated or disturbed during excavation activities, the methodcomprising acts of: A) obtaining sensor data relating to the surface tobe marked, the sensor data being collected by one or more sensorsattached to the marking device; B) retrieving, from at least one memory,reference data associated with a plurality of surface types; and C)generating surface type information based at least in part on the sensordata and the reference data.
 27. A marking apparatus for performing amarking operation to mark on a surface a presence or an absence of atleast one underground facility, the marking apparatus comprising: atleast one actuator to dispense a marking material so as to form at leastone locate mark on the surface to mark the presence or the absence ofthe at least one underground facility; at least one sensor for sensingthe surface to be marked; at least one user interface including at leastone display device; at least one communication interface; at least onememory to store processor-executable instructions; and at least oneprocessor communicatively coupled to the at least one memory, the atleast one communication interface, the at least one user interface, andthe at least one actuator, wherein upon execution of theprocessor-executable instructions, the at least one processor: A)obtains sensor data relating to the surface to be marked, the sensordata being collected by the at least one sensor; B) retrieves, from theat least one memory, reference data associated with a plurality ofsurface types; and C) generates surface type information based at leastin part on the sensor data and the reference data.